


Institutional Archive of the Naval Postgraduate School 


Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations l. Thesis and Dissertation Collection, all items 


1988 


An interrogative model of computer-aided 
adaptive testing : some experimental evidence. 


O'Donnell, Patrick A. 


http://hdl.handle.net/10945/23276 


Downloaded from NPS Archive: Calhoun 


Calhoun is the Naval Postgraduate School's public access digital repository for 


΄ К D U DLEY research materials and institutional publications created by the NPS community. 
| Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 
th 
KNOX appointed — and published — scholarly author. 


"Wy LIBRARY Dudley Knox Library / Naval Postgraduate School 


411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 








http://www.nps.edu/library 


[y 
. 
П 
H 
р 
LI 
P | 
EJ 
. 
. 
" H 
5 H 
LI 
E 
П 
1 
1 
. 
Д 
. 
. 
б 
PIC η 
К . 
πα. ЫШ y 
5 E 
η 
. as 
" A П : E d = B 
E 
. 
LJ LI 
. 
. H 
LI LI . LI 
. - 
un 
r б 
. LI LI Li 
L] LI LI LI ΕΣΩ 
g . n η 
LI ПШ . 
f П 
un А d 
A а 
A H 
η 
LI LI LI [i * 
LI О LI LI LI 
d Е i ДЕ, 
$ 
LI d 1 ШШ of 
П ans 
£ E 
Ἴ AG 
ΓΩ 
П Ы ie 
ο LE 
1 e n б 
LI 
d i ' 0 ы 
H P Й 
А ' LI LI 
P 
П 
б 
0 
Р в E 
ΠΠ А E 
LI id a 
L А E 
Б m 
: ' 
. LA 
. 
D 
D 
Г LI 
E LU ШЕ 
», 
H 2 + 8 x 
P 
з 
r ' LJ] 
а а Я ΠΠ 
LI » 3 
LI 
0 [ED ы 
D 
LI Pu 
. 
" 0 О 
ΓΙ LI 
P . 
' б, 
LI 
" П 
LI LJ . ы 
Й Ρο А . 
П ы i 
P E 
Ж H Д , P ш 
E О 
А e 
П . Ц 
П б 
LI ld ΡΕ LII 
"1 LET E LI L] E 
‚ Г |] 4 ы 
П б А A D s 
0 
И Ae [1 [LET 
D е Й б, 
L] 1 LI LI Li Li 
. bi t ГИ] П LI 
go. x An Й Й . t 
L] E A П E L] 
D " Й Й '*oc П , D 
A D H 
ы Д ' П ο d Ц z 
D А E 
. 
П z οκ " E ' у 
Й 7 Й ^ А 0 ΓΙ ΠῚ 
А F т А Д 0 
Ц Г) ДГ 
П я "oc 
ΠΠ E P η 
L 
t . 
Й Ц a 
П 
LI А " mm А LI 
A 0 * 
E Пех" D i 
" А 5 I 
3 Li ES | , 4 
r s ῃ 
A П я! » 
LI А t : ad 
А б .. А D ως. 
4 en Й D 
H 
" Я П ; τ mE А я 
. LJ LI t [ τ 
E] А , LI 1 LE = 
Е " р! з 
Я ^ . D 
4 
ч DA 
т КТ é 4 r1 о 
- А Й E 
Ц ^ П П E à 0 
П ы И) 
О Ἢ] О 
| ; 
8 ЕЈ ΠΠ 
„э L] Li 4 
» H ue Д ΠῚ 
LI Й L 
LI ur 
LI 
η η L] Ы Ы 
П Й 
Ц "Ne d ERU 
" Й А ә П 4 
- D 1 ΟΠ 
ы A А aa LL 
LI 
ы E П П , р 
4 LJ 
П 1 „ 
.. 0 
Й D П 
à E ῃ 
ш 0 0 L LIO a 
HJ P ъ е D 
9 Р n 
Li , Й Ж LI 
H 
LI ГЕЈ ШЦ ΠῚ 
+ 
\ i { 
a Д LEM 
, А 
К 
, 
Ш 3 0 
П 
П ' Ы ' 
1 


















































































































































































































































































































i o M ss кА ww "4. 9 9 МАЯА ПГ. КРЕ | 
τ... x hd ια. "Ἢ. Жш DEO г 4 ox "nm s к uS. ον Ὁ ΟΝ б rri tse Sq je ο ο ντ νυν 
Р Кү WD р ru » va ^ d 4) | ra ЖИЛ АК ААА АРКЕН Л s 
АЛ à i NT " E r αν, ROT А PROC Күзгү; a CS ο απ МУГТ MALE MIU 
te ШЕ Е 4 a E 0 A ΠΕΙ ἘΠ LETTURE Cae η] S ute pet ey ГИЯ g PA LT LA 51] Ц o 
: И р 0 а [EN ПОСТ ТАП РЕД SENT PI ον ROT Pn Lr 
an τ f ' ' a ЖЫ ТЕГ ҮМ, n oo yee a ed NE ied. CAEN Agr raro Ser A 
; ' an E D D a να. πο το. που νυν LY. омма ГЎЙ ^ 
k E А ШЕ μμ 4. 5 =. [wo CEP νην mM DU va RUD р ч чаони три ЧОХ ч КУ 
. Й Ц б » tfm N ТЕ Л P IM E Ж { ΠΩ͂ 4 14 ^ ULT CN kaap 
T η DELE ET A NL. СЕСЕ У ИК. rw A ERT] ia: 04 reas ΙΟΣ iv wr A RA; very О усуе арт wr 
" 4 LIE af rm] X ΙΙ b. t ws, ° 64 L3 MILII wi err DE Ll] DELI 
η А D [] П 4 i E] PIE NA EI "TEM " › vira EN ΣΟ АЧА md Ak Дзе, тч ΠΝ ΤΝ ed ey d Ls 
> А П Lu τα NW Lm 144 ΠΣ πα” ο ME -CALTIM CU EE M MEE mesna Li 
А А ^ П 4 ©) at sda ν΄ CTE W) a oí! R Pr E AS ал mie e porum = 
Ἐν T а ' Ц P [TN MODELLE PUES Pr E a e cA ча DEPTI mee ile tie ee ann 
А 5 " Ly NU DES I rowel "ED v ecl ЕЕС СЕ ЕИО νη ΣΤΗΝ. 
i А t. "M Ca ον a E eee MET ы ade m» r$ p 
5 a E " B Ф 0 КЕ 0 LU [E TR LT 
[] 7 У e v $5; ^ n Nr TUE sp AEN AÀ id VITE Т Ун redemit ides а θοὴν eer 
y: ΚΓ ET] D rs Μι ο ΜΕ И M rri, мда Ае л ыы, 
g А " "A ` "NE П aha ae Me OW a ee Se Oe CS MSE Ly ORT Wy Piet Bord baad ET μα ее PLU tt 
ES А - " П T Lond PCI ττ.-ᾱ-;:;υ 520 Aviv ^ qd S Ardntet AMAA. iP ie CR er ar er re er AP Tipps 
| Р i D πα OCDE IEEE E Yo T gtn EA το yd qr Pn DH LJ pO More К ЫМ ee seno e 
E Ὁ τ ГАД ` D ds h NET =. 0 НР АЗК Ard n go ODE hr arr bd rd AR DU MES Nw ο БАД ο ΣΩ 
$ E B ^ 4 5 ad ΣΣ 2*4 D a dfl wile Ы * 9. I. И 5 ње. LI 
ΜΠ LL E ELEC C nnn Ке АА P! Lm FAR. peret fear з "re rs ene a уче rt 
Й . 4." з LL D 5.4.4. 5.4. М p La X NY] (αν d Ux Se et ыш: λος а-и η 5744 Saad 
LI Re ΠῚ n r νυ ατα ory) 1 934 ә? 9.4 9). os add MEE | AE мә δι ή 1 "Ao? P te m.,4 ο sinant oed 
Й * s. Д Ew uL d CIC CDL LT LI TO EPUM LS d deem ~ ELDER 
ra n g Й ** * ος T г ЖЕЕ ο. η [qw wb fr ry yh ME Ew mr ^ Lp POT a 
А "NI L "wa Пе" PES БЧ ML υ ИТУ ЧУ Sealy ise EAT LEE WM LE ΣΤᾺ DELI MEM UD EN PA 
0 E! Ке END ы o NET х w^. Д κ ο ο νο ος μή ο ο ο... 
В ΠΠ] Е πμ Ημ τή 75. a I" dl i X Retry tags Ди дарасла 
ee ` „мк ый! τν Ὁ А ИГҮ Чү Т AE Ы ΚΟ ΠΚΕ ý TA a Saree error an 
А А ή а ο) р ІЛ газаа as УСС Р ede. PE uw ЕСТУ нее уа туи: [T Du V Tore: ELLE LU ee 
А ] . ε D LLL S 0 ΠΣ; m T Qe LAE Y phu ο. τν] [em νος. 11 е IL ee 
A б б. д +$ " πλ ολ a Lee oe oe Hg е Ө Ух. ы "E Ph ο. ο bk ee RON 
А ή А “ did і E Ἡ ῃ νο... ο ωααπκττ.ΟΧΧΧΑΣΡΤΚΣΖ [41] du r ARET en, neers bhatt Miner he οι нө v 
К ο μπα, «hé эл ιο en <r ББА ОЛАУ 1 ur, ^ И ЛЕУ P WEN 
б " А . бтз А to^, ICE PRENNE rede ПЕТЕЛ РР Meany aa Җ уут E πλω. ο. ποσα = к келт 
n ГЕ Й LM ete Vast a sf Veda de ty Sets os Me, Брз ИКЕЯ ΠΡ] LO soll 2и ГЕТ | 
р 9 "m 9 T ws EAM ES аа ГИ ES LENS A η. ΕΠΟΧΗ ae IL ye eed d 
. Р : nan" 1 M SPECTET yt a Se eV N КАЧЫ ао νεα ene Ακίν αφ adobe nier 
2 Р ' τω 4 БАА ПОТУ Г νο μον Αν а Т Аа SUE ITI EA XS н 
E а ЕРА " "an MMC et Sr * Yar wy S ο. EU Кыа Ша А ы ο} A TN LETALI ELD PP tyre Кы эу чу ы РЧ, ыле угун 924, 
i ' Л 1 A ΤΟ ΟΥ (ΧΟΠ eae es Wer i 4 4.4 MAL „ДҮ vi ΟΥΡ урсат de. Mies 
1 А ῃ б, ма лл КҮЛӨЛҮ TOPA руте DEAS MAR М е DOPO 
А 4 4 .. я ΣΠ ээм, 2E ae БАЕР ма. gen AK ЕД1 Mr aie ever pete [ue etie aito 
А РУ б ο. Ην ας ο ο η ο ΟΣ oe ILLAE LM MALIN RON αμ ТАХ ЕЗУ of 
E ` “ } ш a Ж η ιο ΚΟ κα pv Ж g EE A tee wir ας Бран 7 кыру 
^ G ` П EN P" D MAL LIEU Cyn Nea e Lapp EU, Iri aid IO ΩΡ »3 
ta H Н E n B ^ Ide LN el a ae ri ТГУ d un e κα 5 30 e: edente ED te ῤη 
D ΠῚ н ‘ [ "2mm ΠΤΙ ate АЕ! ©. LOT TN X KA [PON or m ML a ΝΣ 
Й б А D 3 [Lr DEP D" DLL mE YES PATE HS у reich ate ΗΠ bana NEL 
`% k 4 P » | 444» α τας πα πμ ovo ΓΝ, tnd stat ἐν je κια. 
E А " ' Й А A , LED x Hed tei Dr Bw ta! ур өз ы жы Μα ТЕР ПЕЕ hri ο ων ЗРЯ 
б D Г ‚ à P Пти Й d ae d m MOLD ΤΗΣ νων ος ο ΠΩ 
Ж 3 ы - Г] Жы [ЖК 4 ων. ITO TIPS УУ, 4 ы uw La en. HELL ELT ERAT TY 
> Й r П » | + б ъд Кт Ч TM зе vn ы КСЫ и γα Lem өер А κ ναι ον να 
ια. е | М * П = П P ИТҮНЕ a pare [un ETES bir i. НЕЯ ETI A cee A 29. А Е ТЕГУУ заза ао. ај, 
η " 1 LI t r о 0 445 fe A T ‘a Эд. As A ra P Ἢ λα уула реро $e epa e з ΝΠ inan 
А P 06. ^ ων RET, HCM EN ος DIR à piece rari i ау "^ VO dm Seva ee УК 
. ' n LO. H А D NT E INL τη DLE MN йы: ΣΩ a Има а 
5 D А 3 ; "- Je LE si rm AT ILE ΤΝ eis ρα. pred 2 or IE Кы LT "ira. pate eas Lee 
А А ο. Й au a ' aq felt ы ry τ us κ 0 $- gh END m κα. Tr md An SUB. 
: f ` а о е А, ДА ma o dioe πρ ΡΩΝ Р ο, RA y fy Lm ei Rn zu 
ΓΗ " 3i Ww 1 [4 1 CE E Ve Jt а = RIT M а r r Айз, rm (n BAe a" aT E by, eh] Ae Р; n B DR mr 
А * А i Β А а. iu B *y SIMA Lh Ы ТТЕ, je EI A n iq ΠΤ $i Cw АМ ERA np 
П E A LI p NA Med zn ILI RID M LS LLLA Ў - au » Ы Wer DIL CDI S πα 3 here n. 
Ben Ё z А П g „ + A ue 814. САР. ^ rot WS ams ποῖοι. "ον. DD Ф, CEA x a dr rn de eT ET ER 
Й " cn 1 Г ч,' 4 f 4 ΠΥΡ "85 КЫ " το ЛГА ei DT ΠΝ B πα ο П зуф ῥι πα 7 E Aa: u^ hd RI Sothern еруу ы 
А ῃ EP ^. up " E [EA "ELLE m ЕТ A Edel DD Ue Ram Mii af И ы ийүү o dee 
E ‚* Чуз, ШЕТ * ^ t de ee wA КУТИ | ANE rr PV κ κος ΟΙ ντ. 
. wf И ' Dl dE Ж D ον αυ μας ТИЛДЕ wt ee CEP! ΠΥ бу. арн уана, 
^ Я n Й I» ΗΝ, Ἐν TD (der tar. n y asl INC LL mos d ἐλ з“ йч БҸ Rd dte spo NAP hal poe irc уы РА 
T мля ΡΝ. ΟΠ poa DONI ECTS чое BP pr po [EY M A ο Υψ 
Joc А И D НЫ СУКА ЕТА * ЖИГ, M το, ο Ate a4 Mal eri LN Re: qur S E Er ον 
E n r D ШТ] τ EMT М Ον ON У 5 te ps en (ΗΕ A А pow pm ied Уу aes, 
А : TET б ae νοκ σας [nr P TLLA t e Moe MASA em iit ph dde npe» TEN nas 
neo " H П A m к ie ы р шә 1 АТ мы maar Fein laa ee η COM +h " PAER V sit v a aie et oe: 
Η "T D Г: А Se "ms Agya E on 27 ps » - м. 26.254, 
"m D ea Н m б у ros бз " ғу “" τὰ Nos D ETIN i ολη Н DEPT [m ὥς 5 Тату 
" T E" А © э, EXT v) Иза P φον gy ч t ο. ον αν πο P Prey sede rV 
. E Ш ` Ш - Ы 1 1a*. 1 > h Ps Di Re ΙΟΣ У wi ΠΠ EN] iae Це: (P Essi 
PM и D " ШЕ . Hu: Te ΟΣ “ў 4 сд у ος onarpena 
ó D a a R С - τη we m 44 | ο rn LB REL. EE 
ORE E Ц D А И i = if par sf E D р * I beer ide ΩΣ а, τα αν pet Ies ETE je LEFT. 
Am n E He | ο μπι ο. n FC TIERE EC PI чр πας ty we A ari 
7 r " ^ н РЕ M o P А λος Sn EL re d δν. σαντο, Nes > 
ΠΡ Р * . мб ы E Р БЕ s y ute D D o ad ELT DI ndo лн, ary c αν 
π D , Я Р " Г” * dena ч үз ИЧЕ КЫ, ανν ELLE LL IL EE ON η m Er ^ 
А А Y : uc LE Sage) Es а os фел Дев LP c e Bae: UE TNI EE QT 6: 
r A 3 Р Д Tm ELI Ты d sid 
L Й аы А D LS it "t b el а δον η] ajine ΚΟΕΝ ΠΩ ам E eh bet Y 
я ; Te М ' ο ΜΗ te A NIE Pied end ibidem a ru ARM decia De oae 
е $ ae 5 ΄ П а 2 Y N mn ΠΟΙ ΓΝ ἡ αφ ο Te | 
і ^ TA E КЕТЕ FIM xis А πάω ο πα TT T n 
η L D b. kr PER 9 TEE PA aa ἡ E а d 
Ч ` ve de DURI a Me aed 
П е " s & = 7% - σας M" g πώ ενα RMS ee bd vu τας n EC 
^ Lu ы πα PEDALS: PERLE уд ies eee ΠΟΙ T ο ΘΟ; 
: ha X Τῶν κ Ше! A 
Ξ i ry |) АЧЫК ΠΝ ΠΠ 2 Te! 2% T as МН E A DO F Dr 5 
р Ы | a р Mus NE LL tf REL τ όλ ο IV + E τές, E 
g 4 D A": ЧИЛИК ГУ УЛ M Q4 ar ED h 1 a MM ERE ο И prm ο 
А P i д D 1 rt : Н «і М с "atm vem EUN " UN τ 
р : | 7 a! бы : чад! mn RUIT 
E d. AE us MM CIR OR EIS 
Js s ru TRA от τὰ 
Д М А M a» k ; ος Гу Б >. 7 
ο. | р C ME" ry E: 2 аа ч Т JR A A 
E 1 4 [ ος ME E ' ` Е УЧУ NET ГЕК. $ X Km V ERA x Еа E E. 5 
А , м „ЖА T» ^з АЛО" rae - Я 4 bt. е nad Me “- “з, » 
| А rs " 2 PIOS c ГА ^» ee rA Pi {Я у Ee Bre Peg y ELM ος 
- tas " f kd ο -- фәт = ° д LL EDI J es 
r ο f ` Ш c б 2, Ш к s su ahe ark ος ον. tc ns PM kon чі 
4 λα 3,8 м μμ А 
LI um D D є ГЕЯ AT А tE τη B * 4 4 d. 
Э; р Ν э c "T: ης ΜΑ 98 ИС 7: pm д Б τα ον 
a 0 V NE [E o ri N E E T ы Ы «4Η ELE P E 
er dar n Ud s HA } Б: Е И же pu y ED „Чер " m A E 
. в LI | i H = 
: "- | IINE ó ME ио SN 
. L] L] Г] η ar LT LP G ο a » s ΠΗ * 
" Jn L ce БД ἴ y = πίη. Or y ы: “к Us 3 z d Tu ΑΜ wh Le be! 
i. . Ш -. ὯΝ | ENDE "m p А н LA „у ee, 0 AE КЛ + ~ 
PLI L * ο "4 m er г“ П ШЧ Pi p ρ EI 
* ' S . п E Π.Ι, У. ў \ "a: η OP REA EH nue nt didis. 
+ EI Ш Р б, Ы EU uU NE UD re x. t у А 
F t П "OE } A DE ΝΟ, ] [PT Cd x Φ'» κα 1 Е T à А) 
᾿ a D LP + ы T : S Kr E n , UM ON е“ E E. к à М Е = ' 
1 . E Ц ГАУ b | 1 ЕЗ v? ^ 
n. А ы » WSSg tm. UNI АГ τν - «ο 3 к 
? ΠΠ, Р ο. ' mit ek Ae, AC 
. П Dan 0 Е bi. АР E 
А k, 
v : 96. EA [ΣΎ 
" 107 П "Wa s [| - ig 
ΓΙ vi e 
Й D D P 4 B ἃ; 
LI * ΠΠ 0 4 στ ή 
В А ΠΠ (d А H MED 
8 x* . П 0 ч ee] Н 
.., να tM E y 
LI LI ΠΩ П П . 
A aoe А ПИЕ ka 5% 
А Ы Т pt 4 
А P T А E © › nme 
i LU LJ H 
Y н ч ον » ΠΠ E ' Ж бА T 
de " ' MS 5 ] Ы (ary "E 
LI D L] Lon] * 
П " at - Я p ELT Τεν 7 PA vir AD. 
9 пат. а ай 1. i wae 
t - "0 "A eet 
* α ы wens id bs 4 Lai Sa p 
З FAS р Й L ΕΣ; 14 dud "e 
mw Р "n П "men П [m 3 κ [PEDE E EX n Pf a 
: νον '' , D us ἘΣ ΩΣ (WE ὦ -- 
E D i. Ετος "4° MELDE CORE ay ч : Ма" = να Е m X 
А " P 1 р П g η б П "E Й б Ne = ΓΝ ΗΝ ET E P ee 
5 А П е t). g mH De T Di OE RE AEST ως T i Ж "> 587 13И 
б А P & А | к & D 79 B "nem Poe 10200 rà bod ul MS PE! LE 
А б А n i 1 P . L s 0 СР ГИ - 
Н Р a э эз LI r! D is Й , Ы У + 0 HET Ἢ ie Sj he ЧЕ 4 
n P 0 Й 4 ῃ А 4 á Й РА? "n Б 5 = rm 
И 3 g П 3 LE : Й П б Pd Je ΡΝ Ἢ " 4 "Nr m . . νυν кер 
КЕТ ΠΠ "> . DID Tua ο η IM | et pr. y'o DT] Ην Ès ap 
0 i { ПЁ ДЕ -- е Des c Ad ο η D Н Й E ολα ἡ "re 
0 Ц Ц τ . Pd LI LI . av. ore 
А ΠῚ 2 rt) D d QE E [EP ТО! “ . » А eo Sirs 0 , {7 К, cre Ф aa 3 E" HE 
"NECI t. ... 2 D bey ‘ А UNA ш να) d ч НЕ oe = 
FA tL on vee ee gt x e Lys 
> П 29 Τὰ ΠΩ κ D 
E 4.27 Ш bL. я 5 aD A ogo ut 
: ΠΗ ҖЕ гө 2 re 
з Г ЫР ee ; pp - » [d Жы ked E. з» к 4 
^ т. 1, АГЫМ: - 
П a ИП J9 "52 TA Lee ὮΙ 1 wr. pow Eat, 
ene ΠΠ LT А ‘ ἘΝ Á LH EDS 2 £s Ps nF 
P " re И τς on T pua eid 
^ А 4 LLLA 1. >. e M m 274 n r 
a Babe ef + a te κ op 
i 2 a? μ.ο. UE ne aur Dip d 
; H "з А ^ es τ... ΟΠΣ ye ΠΠ PE Ih б di M i PEE JA. t ө 
ые " "um дэй л» cs» o, k^ Hu px ee Freie ы ЭИУ Ln Б, Fir πα 2 
' πως n А a ῊΙ PEDE La PLA eye 
А Р А r boo 
RT А FL UE " MH E s iy 
M 2 ". B Li P b τὶ H Н 
hr IET MS ў 4 \ Г А у Ж yeu: m ny А NC h 
Ρ A e M uc LEES [Ls M 
ta rs Й б „ i ween Й La B . i ПЕ "UN OTEA ITEMS a o 
’ é А ^ ба ч P Y] m rA + бе Ч: ee tee “he, ИТЛ ИЫ А E^ 2 р Jy Ὅρα БЛ» рак {5 
И ELA f 18:1 б ET б " t.. к no» e А DE is Ἔν б T "LE «ΧΑ E paper 
j : ш 0 PE Е E "E "өү к, Y 4 Eon CE ар M 
t [1 oa "i + { ad per H LE idu "om 
= ΠῚ П П э, om Й e ses CET d En 5 κ H 
^ і of + КУ) А PE 5, ША "р PE ү} И ine LL A wen А Та p -| 
p E " 0 б se k П ч X E E [^ Ho 
E бозу Й э а à 7 ” СГ IL «m "„ 1 Ж АЛЛ ӘК ЕТШ m E ча з LED DES πῇ 
| i "VU o MES et б б ЗАНАТИ M ας Н Pome sr el fy 
r eu be PL О Le у а F р i num ΠΡ A ae Πλ Re τί уы PI NL LTD Жыр Bm "at goo 
" on LE of А EINE П ΠΤ nn E mb "un i τμ as 5. Π LM Шү αμ. μοι Dg ee serial f. ы. ата eI. 
В D E LE ae NIU T һҺ 5 ai ог ΩΤ η VEM e PS, ТИМ) DID hs du m does Ни Por PX ЫЧ 
. E РЫЙ EU + + DOE ο] P, 7558 the 4 us v pe SS my Ge aN Ph PAN κ ΤΗΝ p RC ж ЖШ 
А | A Е 7 Е АС ΡΕΜΠ е Yu ΠΤ Der rd DNE D Zum ШУ Mabe C Pa Pan en Mut ae E Ж А η τμ M. wer, iy 
"n А [7 за А Din 5 ^ ENDO A TEE. m ΠΠ an "1 VP APD RERE Ai E ο e*a pis alie P 
; : TIO i πι ТООЛО c 6: У ο χω б Е ΠΠ ΗΝ em Р 
Й 1 0 ПЕ з πέν rg J 4; 49 > - 0 L] SIL ы 0 κ 0 Ee Pus П ΑΝ . тен ra T 
435 "Tn E P biro er ΠΠ 5 ΠΠ D [ELECTI pu L is NS 05 DE "ou "ч Ц IE [YD UA LEE ών ПРУТ 
^ " 5 i a ΓΙ 19 g ΠΠ τ A ' . А П П . Κο [ΠΠ 4 fr fU ПА? 
И К А 1 Й ТИТ ΤΙ PP ОСТА, Red a ZA П TH E nr. [m L o4 »* hii ΠΠ AARNE + {з 2, аи P 
А , а А ΠΠ А ПЕТ ЕТ mE 3 PR TOC NC n Us ru ruik δῦ ts E du FEE LOT ος 
Ш ШИГ 4 ' б, Beh Sok Te Jt ЕПП ели П ο RN LED NOE DUE T n ον pa a vult j- Mov 
" g ve E Й Па О 0 00 " LEE] hi de cC CPP A ΤΙ 
“a Й Ü б, D NL > DCE te η Os Me on μ > HENNES d i ire узр ΠΠ m 
* P D ΠΠ 14. CT | ΤΗ DR LL E "а E 1“ UM ο Gene genta a> wy et nthe 
4 I TT оо оо Ост ЖЕЗ ο ο ου σα τα са we jb d d d ATA μαμα ΠΩ eti ΠΟ 
И 0 ΠΠ rm П . 1 . 1 ИИ ee Jn КТ Hi P At фе ТИСУ Pe at ap ahs, ate 
Pa ripe ub TRO 144: ҮК ο OLET ё а АГИ ΟΠ оэ n е fa Ἢ n A A ТЕГУ LIT Р 
D DON. Lom ῃ A ПЛИС б а os YE. ie "n T] m. η M ΠΤ DOTT qoc ην 
‹ ' СУ ο. a L Р è Д sins” raed a, мн he ΜΗ CORT CLF DE ye 
Ων] ο μας ποσο лы оО Ы ο NP μμ Ιώ μη у DEZ EM см 
Le εν πρ ο S" ОСИ У ЕУ ΓΣ ΠῚ E og ra μα. p ση ке сады ^ A 
708 ΗΠ E ПЕД) . 0 πα ΠΗ PCS Ma es ПК 5) MGE эл з'ем PED n “Ἢ 
Т A А fer ce A ЫП eat ” о КИИ У Ы) ш D ια ο ΤΝ A „кё ΟΠΣ БКТ $e ШҮ H 
a " 4 DON eer DERE ΠΠ ΠΠ EN LSU t* ΠΠ ΟΠ; tad by ГА. nm ΠΠ Р απ ος oe μετα po M τα 
2 ΠΠ πο, ου JS cc "rs ΤΝ E MID E d B. ΑΟ МЕЧ πο Re ae 
PE ο П ОРУ V > e riy Π τ] ΠΠ αὖ" ΄ 4 γν ἌΝ τ Κο αμ χα nre DOE n БААСЫ er E κ τη 
Επ n A ieee iia. m ЕИ ог £ Γ΄ ΠΠ ER ТАК, БАШ 435} Hn RE TUE ΠΕ ΠΡ at ad y ni ΤΟ A 
n D. п (OM [E LE CN PP IP E: ms DABIS LM T LEE M n οκ чї; ru СС Ep T ДЕМ ΠΣ he we abet mo 
ЯС А DOE ο D ДД ον ο Ж Ж COUTE εν "ү πο. ων ρα. mun ο ry DNE M CHA Ap P ΠΤ νη уд eta y ten 
a . , n ' DER DD fae С D 0а "E DEAD E ru ча 09. и, IN ΣΙ ΟΥ E PL aT SP TR Le Sie ren РҮ ҮН ὙΠ ΚΠ РУ 
ΤΙ t’ PP 7 D ci πη LE 0 «Ау рз ip La CO DS Е ГНН t. DX abe do р ΤΩΣ η "t Lr СИРР 
"n " E А E EN - ТЕР р Б ΠΝ ΣΣ Ns эө. TE D! Е ΤΑ un E] н Ш “ Nee ы PES TU gn 
ο = s ο Πα УА "ирд т ае pt ο ο ο μα ο η ΠΠ ὀχ" κ [LU ME "T tyr c-r o φρὴν oar AA Dr 
as С εν κι ο Τῇ en Ы СИОТ "ng чей ο Ay pfi ΟΡΟΙ μα Μα Т eo vns P iota. 
η 4 ШЕП DIU UD DE sve LG NL D d. Ὁ D АЛАС КҮЛҮ КЫЫ HE DC ауа ο ΕΕ Fre e „ 2 a Αν 
ΠΠ В П П πο. η DE ex DEM ПЕС tA νο АРЫ 3” ΝΘ ΠΤ e Гоно > p 
^ А Ц ο. . О ё м Ы Den ш UR " 9 whe 4 8 PI р C Ha D u ve 7 Eo "у Ач " ΤΡ Sto ae πα PET 
A Е Я ΠΠ " Й ΠΣ τη . П 4 LEES Td epe та» ШШ νην ΟΊ ΤΩ παντα ШЕ a LE LL PP ote АГИ fe 
H 4 LRL ια т. aNd i et 4 eet Je] at i Y» ACD ΤΙ AE К-ДА μα . “ae 4 LIT MEC P ro Pa es 
^ ’ 8 E H DT nU ν΄ ΡΨ TL ΠΧ Qu ҮЛ or in en a S NE Be eL ΠΩ PM ag LL ILE PRECOR X ү 
κ. aG JO Й D АТАУЫ d por UE YU d p EU POLES au E NC ο Wei sk ΚΥΠ ο ΙΣΟ TT а. 
С, А r | E П ^ H КАТЫ: LAU ΠΠ ү $ ACE Di ο ΕΟΝ DH TET lI ED M CP t 
T Й СЕС И | E r П DINE ME UID ILES ZEEN REL s NEN К dil ΠΝ ΕΤΩΝ тю race Cri Tae ΧΩ HISP PU a ee Po ee eee И 
А ge ΠΤΙ. Da Ule bU QAO" nw CA e КЕЛТ Ck Ck TE yc Par OP MED - ο πα HEU EA Sil a e ΕΣ 
5 ολο κ ЫЙ}, ЧЕП 2 1 а ШК = yo rai Lu | A CT d ος {, 495 ΗΝ. LEM d ys хо аа Т Tí мы κ E EE DUM 
О D J z Я p "aa ы И Eu οι 2 d Π LP m 
- А ΠΩ σα, 0 б П is τ LG L Im 5 [4 EL [2 Los LP LS МУНЯ уі DICITO M ГЫС pus tede, do ky YE cutn 46 
WS n " П ΟΠΠ Siok τὰ ΠΩ ΠΝ M i bons poe Vp, dE per HB Hp Se ГИ ТУРЕ еы 
А Й "n" 3 mo . n A «p К. " D a й » х SA an τ п an ant d Μος. = FA T» bdo ο.” ae ΠΡ [Μο lo momen D 
Й .% ΠΠ СИ 158. ААА j “а, ОНЕУ ТИ УУ oti κ η IT Ц 
E p E П va ы Ц L Ec X: ni E nets р νε... rg et? CLS P κα ΠΤ OI t WS b ау те ee Ды Аа 
A ' PH { m M %% аз ΠΤΙ αι ΚΟ T Dr ee πρ ση τοι ee E 
» c Й EE E E ПШ m LL Ed Ἢ δ. ni lu NC ELE" * ru Dn шы МАМ ДЫ | ae ΠΟ A μμ do dv s uM 
ο η LN ΄ E Eh. UEM ОРАМ ^id ' bove Ae gia wur нна ΠΡ. ОЕА С АТАА eee 
rs i D- EIS; a ο yea in A ΜΑΣ) Los A ein LE EE μὲ 
A. C 4 E ? as 1”. og "^ ἐν" ἐλ. Λη и ЕТО E УУУ t£ неч [е Е-е 
А i Ἢ T o Ὁ. Ὁ д; | A CEPS IO ΠΤ DM PORCH е 
4 " "e A N bL! ΠΝ i D ы ΠΣ E at СХ а АТАЛА y 57 τν Aeon ` A E a ТЫ E БА НЕЧЕ 
= A б» 8 a mou t CE M P G Ш D esr t'es. 1.4 abt Ja CE TA gs » aD ry q VS cad D AE AME ο μα ο Ὃν 
ў E : " fei κ ΠΕ sS Ў "су. My ЧА ТЫ ПАДНА tee 
CAT: an r DES CM νο} d E 1 e - 9» Fat; is est hy teat eel απ η AE E "e I Tu DN Ly DEDE 
Р | JOE E M SL " d у Бо, AK Uc M s eoa Proh D oru s" prety ο μασ “Ж, 
К а , Ἢ soe П ο. πο ΩΣ I б Ў. ML a Hi О "DUE κ πε” ιν "^ ЖАН ΩΝ; ope (ДҮ; Nus ELM! A ^ 
А р Ρ " 4» 0 h We T EJ Г. АСЕТ Zh MAL e IS ΚΤ μὴ pe ενω τὰ ο άλτη, ae 
, "M n а i E IU T aW. να αρ Я ёа уз эин ЭН ЫС Ы ERIT ДЫМ ΠΠ ΟΛ Α VALOR E eere 
А б А Г . PLN. П 0 ПЕТ Ц 4 Li О vh Ч П “23 OF E al Ы 
d : ё А é ОДЕ ДОСТ АО κ СЧА аА р „Аш = n Ly ри з σα 
| , EM DOCET ius "T ο Ree ste et, Np gente, eet. Monty jon OST D Feel Ai DUREE ina σον 
GC " to А arn ie em κ TP ж S 4 wt ff d лт [ E Жр D nd ALD T ЯТУ "mari AIRES I. ' Е pie нк (Не 
0 E " Й П П i 0 Dd u П “ч a wen « t^ A 
Jae M J TEE ЫЫ ы Оса ВОЈ P a τ ЖЛ Ш vu PE E qu iw TEN on uc П E 11] τ δα η de da КЕЕ а ННН 
ο κος USE IP D TE TP. ΠΠ a pee on si pert Hee aaah eer pote 
' Per i ПЕ 1 ON Qu ΣΝ τν NL Be Qu Us D UO 2 + эре ү Dd Jc ΠΠ MON rer Jac NA АЙ А '. γον ΣΙ ως 
E ' oF [AL р 3 saa ΠΩ . ae LESS | "T t r a t П а дан $ 2 
Jen δε, „í Mur OI $& w Ш ES. E Ν΄. x Ju fi У КЫ be TAE "reg "E PANE b Н na KI ΟΝ yeh he MP 
E ΠΠ 0 Er qoa . [AME n. Li "s бе ee УУЧ". б La D [17 LE r u unu 
APT Й Je п Е E А R Р pa mer LO DIE КП fi m СТЯ Me 4 ww ТАБАДЫ Ө я” ΝΤ ΡΟΗ eir КЕЛБЕ z De bars Lir Е) П аА 1 
а x П . ΠῚ m Г ' {Py 
. we r3 E 














NAVAL POSTGRADUATE SCHOOL 


Monterey, California 





ο р | LAN 1 
te^ ( 


THESIS 


An Interrogative Model 
of Computer-Aided Adaptive Testing: 
Some Experimental Evidence 


by 


Patrick A. O'Donnell 


& а 3 


September 1988 


Thesis Advisor: Dr. Taracad R. Sivasankaran 





Approved for public release; distribution is unlimited 


T242213 





E | ASST "aTION OF THIS PAGE 
o 


REPORT DOCUMENTATION PAGE 


ES AT SECURITY CLASSIFICATION 1b RESTRICTIVE MARKINGS 
Unclassified 


в SECURITY CLASSIFICATION AUTHORITY 3 DISTRIBUTION /AVAILABILITY OF REPORT 






Approved for public release; 
Distribution is unlimited 


|} PERFORMING ORGANIZATION REPORT NUMBER(S) 5 MONITORING ORGANIZATION REPORT NUMBER(S) 


р DECLASSIFICATION / DOWNGRADING SCHEDULE 


60 OFFICE SYMBOL 
(if applicable) 
Code 54 


ja NAME OF PERFORMING ORGANIZATION 7a NAME OF MONITORING ORGANIZATION 






Naval Postgraduate School Naval Postgraduate School 





€. ADDRESS (City, State, and ZIP Code) 7b ADORESS (City, State, and ZIP Code) 


Monterey, California 93943-5000 Monterey, California 93943-5000 





a. NAME OF FUNDING / SPONSORING 8b OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER 
ORGANIZATION (if applicable) 








c. ADDRESS (City, State, and ZIP Coae) 10 SOURCE OF FUNDING NUMBERS 


PROGRAM PROJECT TASK WORK UNIT 
ELEMENT NO NO NO ACCESSION NO 
1. TITLE (inciude Security Classification) 


AN INTERROGATIVE MODEL OF COMPUTER-AIDED ADAPTIVE TESTING: SOME EXPERIMENTAL EVIDENCE 





2 PERSONAL AUTHOR(S) 
O'Donnell, Patrick A. 


3a TYPE OF REPORT i30 TME COVERED 14 DATE OF REPORT (Year, Month, Day) {15 PAGE COUNT 
Master's Thesis τα. τοπ 1988 September 101 

6. SUPPLEMENTARY NOTATION 
The views expressed in this thesis are those of the author and do not refelect the 


official policy or position of the Department of Defense or the U.S. Government. 
7 COSAT: CODES 18 SUBJECT TERMS (Continue on reverse it necessary and identify by block number) 


FIELD SUB-GROUP 


Adaptive testing, Computer-aided instruction, 


Bayesian statistics 





9 ABSTRACT (Continue on reverse if necessary and identify by block number) 


This thesis proposes and validates an interrogative diagnostic model of 
computer-aided adaptive testing. The model uses an alogrithm based on Bayesian 
techniques to determine both the number and the difficulty level of questions to 
present. The power of the model lies in its ability to automatically adapt a testing 
session to the individual's level of knoweledge by choosing the domain, type and 
number of questions. The model was implemented as a computer program and was tested in 
an experiment with 34 military officers who were masters degree students. Analysis of 
the results indicated that the IDM can evaluate student knowledge as effectively as 
traditional written examinations but with a significantly less number of questions 
and shorter test duration. | 


A DISTRIBUTION; AVAILABILITY OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION 
| KJ UNCLASSIFIED/UNLIM:TED 0) SANE AS RPT C] DTIC USERS 


ta NAME OF RESPONSIBLE INDIVIDUA- 22d IELEPHONE (include Area Code) | ¿2c OFFICE SYMBOL 
Prof. Taracad Sivasankaran (408) 646-2637 Code 54sj 


| 
DF ^ 2 a3 APR ree $4 . ΠΣ 1 T 
if OT t 
DI ΕΟ ο ο €t U.S Government Printing Office 1986—606-24. 


UNCLASSIFIED 


Approved for public release: distribution is unlimited 


AN INTERROGATIVE MODEL OF COMPUTER-AIDED ADAPTIVE 
TESTING: SOME EXPERIMENTAL EVIDENCE 


by 


Patrick A. O'Donnell 
Lieutenant, United States Navy 
B.S., United States Naval Academy, 1982 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN INFORMATION SYSTEMS 
from the 


NAVAL POSTGRADUATE SCHOOL 
September 1988 


ABSTRACT 


This thesis proposes and validates an interrogative 
diagnostic model of computer-aided, adaptive testing. The 
model uses an algorithm based on Bayesian techniques to 
determine both the number and the difficulty level of 
questions to present. The power of the model lies in its 
ability to automatically adapt a testing session to the 
individual's level of knowledge by choosing the domain, type 
and number of questions. The model was implemented as a 
computer program and was tested in an experiment with 34 
military officers who were masters degree students. Analysis 
of the results indicated that the IDM can evaluate student 
knowledge as effectively as traditional written examinations 
but with а significantly less number of questions and 


shorter test duration. 
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I. INTRODUCTION 


In education it is crucial to access objectively and 
accurately a student's level of proficiency. Computer-Aided 
Testing (CAT) is becoming one of the most effective 
evaluation tools available to the educator. The greatest 
challenge to developing computer-aided tests is modeling a 
testing session in an intelligent manner. 

Most current computer-aided tests are based on 
enumerative or adaptive methods [Bejar 1986, Thomas 1986, 
Larson 1987]. Enumerative testing is similar to traditional 
written examinations in which an individual is asked a fixed 
number of predetermined questions. Adaptive testing on the 
other hand selects both the number and type of questions 
presented to the individual according to his personal skills 
and previously aquired knowledge. 

This thesis proposes and validates a computer-aided 
testing model called the Interrogative Diagnostic Model 
(IDM). The model is both declarative and adaptive in 
nature. In other words, a pool of questions are explicitly 
stored іп а test bank along with their difficulty level. 
The model uses an algorithm based on Bayesian techniques to 
determine both the number andthe difficulty level of 
questions to present. The power of the model lies in its 


ability to automatically adapt a testing session to the 


individual's level of Knowledge by choosing the domain, type 
and number of questions. 

The model was implemented as a computer program and was 
validated in an experiment with 34 subjects. Analysis of the 
results indicated that the IDM can evaluate student 
knowledge as effectively as traditional written examinations 
but with a significantly less number of questions and 
shorter test duration. Chapter II of the paper reviews 
related literature in the area of computer aided adaptive 
testing. Chapter III desribes the IDM in detail. The 
subsequent chapter discusses the computer implementation of 
the model. Design and administration of the experiment is 
described in Chapter V. The statistical analysis of the 
results are presented in Chapter VI. Chapter VII concludes 
by stating the advantages and disadvantages of the model and 


Suggesting directions for future. 


II. LITERATURE REVIEW 


Although considerable work has been done in the area of 
computer-aided instruction, research in computer-aided 
adaptive testing has been sparse [Brown 1977, Clancey 1984, 
Clancey 1986, Davis 1983]. Weiss [Weiss 1977] described a 
number of strategies for adapting test items to individuals 
differing abilities. Tennyson and Ruthen [Tennyson 1977] 
verified that an adaptive strategy in selecting the number 
of instances needed to learn concepts was more efficient 
than either partially adaptive or non-adaptive strategies. 
Further evidence that adaptive testing has validity was 
presented in an experiment by Thathon and Kruawan [Thathon 
1985] in which a self scoring, flexi-level adaptive testing 
procedure proved both effective and practical. 

Bayesian statistical techniques have also been 
investigated in the adaptive testing. Sympson  [Sympson 
1978] applied Bayesian techniques to measure ability of 
Candidates using a hypothetical 20 item adaptive test. 
Bayesian analysis has also been applied to illustrate the 
basic mathematical concepts involved in scoring multiple 
choice achievement/aptitude tests [Warm 1978, Vale 1981). 
Further application of Bayesian analysis to adaptive testing 
was done by Sivasankaran and Bui [Sivasankaran 1986]. Their 


work provides a basis for the IDM. 


Methods of interpeting a candidate's response and 
immediate modification of the test itself were discussed by 
McArthur and Chou [McArthur 1984]. Loyd [Loyd 1984] 
presented evidence that a two-stage adaptive testing 
procedure, in which the first stage routes the individual to 
second stage which focuses on the individuals ability level 
is amore precise measure than a single test. Bejar  [Bejar 
1986] describes a further development in adaptive testing in 
which the testing program not only decides the number of 
questions but also actually generates new questions based on 
the ability of the person being tested. 

However no attempt has been made to investigate the 
effects of incorporating learning theory into computer-aided 
adaptive testing. The interrogative diagnostic model 
presented in this paper makes use of Bayesian techniques and 
incorporates a multi-stage testing procedure based on 
cognitive learning theory proposed by Kibler et al [Kibler 
19701. The experiment conducted using the IDM was patterned 
after the design of Thathon and Kruawan [Thathon 1985] in 
that a group of individuals was given both a written and a 
computer test covering the same topics and the results of 


the exams compared. 


III. THE INTERROGATIVE DIAGNOSTIC MODEL 


A. INTERROGATIVE LEARNING MEASUREMENT 

According to the learning theory proposed by Kibler et 
al. [Kibler 1970], an individual progresses through 
different levels of learning (Table 1). Levels 1 and 2 form 
the first phase in which the individual begins to recall or 
recognize information and be able to translate it from one 
form to another. These two levels draw heavily on the 
ability to memorize information. During the second phase of 
the learning process, levels 3 through 6, the individual 
assimilates and integrates information of increasing 
complexity. 

The IDM has accordingly two types of questions’ which 
directly correspond to the these two phases of learning. 
These are the Supplementary and Critical questions. 
Supplementary questions are those questions which are basic 
in nature and cover levels 1 & 2. Critical questions are 
those questions which are advanced in nature and cover 
levels 3-6. 

The rationale for the Interrogative Diagnostic Model 
(IDM) of testing is based on the dynamics of an oral 
examination. Before an oral examination, the examiner 
prepares a set of topics to be covered and within them 


identifies specific questions. But the difficulty level and 


number of questions actually selected and presented during 
the examination are guided by the student's answers’ to 
previous questions. This type of interaction can best Бе 
modeled by incorporating Bayesian techniques. The IDM makes 


use of these techniques. 


TABLE 1 COGNITIVE LEARNING LEVELS [Kibler 1970] 


LEVEL (Lowest - 1, Highest - 2) 
1 KNOWLEDGE--recall or recognize information 
2 COMPREHENSION--translate it from one form to another 
3 APPLICATION--apply a known procedure to a new 


situation 
ANALYSIS--break down information 
SYNTHESIS--put information together 
EVALUATION--evaluate something according to some 
criterion 


с (л 


В. MODEL EXECUTION STEPS 

Figure 1 illustrates the model in the form of a 
flowchart. The model first formulates a hypothesis about a 
student's proficiency in a given topic area and then 
evaluates that hypothesis using a bank of questions. The 
model then derives conclusions about the student's 


knowledge. 





Figure 1 IDM Flow Chart 


The IDM consists of eight steps: 
1. A topic area is selected. 
2. A hypothesis is made that the student is more likely 
an average student and a prior probability value 
initialized. 


3. A question is presented to the student. 


4. The student response is evaluated and the earlier 
probability is updated. 


9. The probability is compared with evaluation criteria 
and further questions are asked if the criteria are 
not met. 

6. The hypothesis is accepted or rejected. 

a. If the hypothesis is rejected it is because the 
student is either above or below average and proceeds 
to step 8. In either case the model selects a new 
topic area in which to query the’ student. 


b. If the hypothesis is accepted it is because the 
student is average and the model proceeds to step 7. 


7. A decision is made to: 
a. end the test if all topic areas have been covered. 


b. select supplemental questions if the critical 
questions have been asked. 


c. select the next topic if the supplemental 
questions have been asked. 


8. A decision is made to: 
a. end the test if all topic areas have been covered. 
b. select the next topic area. 
The Bayesian theory as applied in the interrogative 
diagnostic model steps shown above is presented in detail in 


Appendix A. 


C. BAYESIAN PROBABILITY AND QUESTION ORDERING 

Since Bayesian probabilities are affected by the ongoing 
performance of the individual, the ordering of the different 
questions selected during the testing process can 
significantly affect the judgement about the students 
proficiency. Three general methods of question ordering are 
considered: decreasing difficulty, increasing difficulty and 
alternating sequence. If questions are presented in a 
decreasing order of difficulty, an average student 
encountering the more difficult questions at the beginning 
of a testing session might be inappropriately underevaluated 
since he would reach the cutoff threshold probability early. 
On the other hand, if the questions are presented in an 
increasing order of difficulty, the average student might 
tend to be judged brighter than he really is. A third 
method is to order questions in an alternating sequence of 
difficulty. Theoretically, several other methods are also 
possible. 

The effect of the three orderings on the student 
evaluation is’ pictorially in Figure 2. The first graph 
shows the effect of a decreasing order of difficulty. From 
this graph it can be seen that the moving average of the 
difficulty does not move down as rapidly as the actual 
difficulty. Regions I, II, and III of the graph represent 
the level of difficulty that a poor, average and superior 


student could reasonably be expected to handle respectively. 


—@<@— - (D <0 ~ 
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Figure 2 Effects of Different Question Ordering 
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In the first graph, the average difficulty level does 
not fall into the average region until the fourth question 
has been asked. By this time the Bayesian calculations may 
have already concluded that the student is poor when in 
reality he is average. 

In a Similar fashion the second graph, showing an 
increasing order in difficulty level might overrate an 
average student. In this case the average difficulty level 
is below the actual level and only when the fourth question 
is reached does the average difficulty level move into the 
region where the average student would be challanged. By 
the fourth question the Bayesian calculations may have 
concluded that the student is superior when he could be 
average. 

The alternating sequence shown in the third graph 
overcomes the limitations of the previous orderings by 
alternating the question level. As can be seen the average 
difficulty level quickly moves into the average area and 
remains there with ever smaller oscillations. By remaining 
in the average area there is less likelihood that an average 


student will be under or over evaluated. 
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IV. IMPLEMENTATION OP THE DIAGNOSTIC MODEL 


Implementation of the interogative diagnostic model was 
divided into two main areas: program design and question 
design. Program design was closely allied with the first 
assumption and Bayesian statistical techniques presented 
earlier. The area of question design corresponded closely 
with the second assumption that questions can be written 


which test different levels of the cognative domain. 


A. PROGRAM DESIGN 

The design of a program to implement the model was quite 
straight forward. dBASE III Plus was the language selected 
for implementation since the questions were easily stored 
and manipulated in a database. Because the specifications 
and structure of the program were not fixed, a prototyping 
approach was used in designing and writing the program. The 
final program consists of eight modules whose data flow 
diagram is shown in Figure 3. 

The MAIN module retrieves criteria to be used in the 


Bayesian calculations, opens data files, selects topic 


areas, and calls subordinate modules. The INFO module 
presents the test instructions and gets student 
identifcation information. The ASK module presents 


questions to the student by calling the DOIT and QUESTION 


12 


- ΓΞ 
DOIT CORRECT QUESTIONS 


Figure 3 Program Structure Chart 
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modules in the case of critical questions or from data files 
in the case of supplemental questions. 

The ASK module also forwards student's response to the 
CORRECT module for updating posterior probabilities. The 
FACTOR module calculates factorials for the CORRECT module 
and the TALLY module writes the question identication 
number, student answer and time taken to a data file. 

The program evaluates the student in each topic area 
using the Bayesian technique presented in Appendix A. The 
numerical scores given by the computer program for each 


topic range as follows: 


5 Superior 

4 Above average 
3 Average 

2 Below average 
1 Poor 


A pictorial representation of the scoring process is shown 
in Figure 4. The student's total score for the entire exam 
15 an average of the scores in each topic area. 

Upon completion of coding the program was compiled using 
CLIPPER. This conversion allowed the program to execute 
appreciably faster and eliminated the the necessity for the 


student to begin the exam by first invoking dBASE III Plus. 


B. QUESTION DESIGN 
In designing questions consideration was given to 
question type, content, cognitive level, ordering, and 


presentation. The types of questions used in the computer 
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upper cutoff reached 5 
student is superior 





lower cutoff not exceeded 
student is below average 


Figure 4 Computer Scoring Process 
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exam were limited to multiple choice and true/false. Due to 
the inability of the dBASE III Plus in processing natural 
language essay type questions were excluded. The content of 
the questions came from the textbook instructors manual, 
past examinations and the instructor. The questions were 
catagorized by the instructor into critical and 
supplementary as discussed in Chapter III. However, instead 
of assigning a specific difficuly measure to each question, 
each question within a catagory was assigned one of three 
difficulty levels, with Level 1 being the least difficult 
and Level 3 being the most difficult. The text of both the 
critical and supplemental questions used in the program are 
presented in Appendix C. The alternating sequence of 
difficulty discussed in Chapter III was used to order the 
questions within a cognitive level. 

Presentation of the questions to the subjects was 
challenging. Many of the critical questions were too long 
to, Shit on the standard 80 x 25 character screen. This 
problem was solved by partitioning each critical question 
into two program files. Dividing the critical questions 
into two screens meant that the students were required to 
page back and forth between screens to view a single 


question. 
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V. RESEARCH DESIGN 


A. SETTING 

A micro-computer lab equipped with 25 IBM PCs was used 
for administration of the computer exam. A diskette 
containing the compiled version of the computer exam was 
given to each student. A week prior to the actual exam a 
demonstration exam was given to the student in the same lab 
in order to let the student become familiar with the testing 
program. To further ensure that the students did not 
encounter any technical problems a facilitator was present 


during the examination. 


B. SUBJECTS 

The subjects selected were masters degree management 
information systems students enrolled in a database 
Management course at the Naval Postgraduate School. The 
subjects were of a homogenius background; all were junior 
military officers in their third quarter in the computer 
technology program The course was divided into two sections 
of 21 and 13 individuals respectively, and the same 


instructor taught both sections. 


C. EXAM DESIGN 
The subject area, database management, was divided into 


three topic areas, basic concepts, advanced concepts and 
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methodologies (normalization), and issues related to 
management of databases. Each topic area had 5 critical and 
20 supplementary questions. Both the written and computer 
final examinations consisted of the same 75 questions (15 
critical and 60 supplemental) and both had a two hour time 
limit. The written examination presented all 75 questions 
to the subjects and the questions were not grouped according 
to topic area or cognitive level. The computer exam 
presented a varying number of questions to the subject 
according to the Bayesian evaluation of the subject's 
previous answers. In the computer exam questions were 
grouped according to topic area. Upon finishing the exam 
each student vas given a questionnaire to determine his 
reaction to the exam. 

In order to compensate for any learning dynamics which 
could have influenced the responses from the first test to 
the second test, the experiment was conducted in the 
following manner. One section was given the computer-aided 
exam first, followed by the written exam. The other section 
was given the written exam first followed by the computer 
exam. 

Upon finishing the exam each student was given a 
questionnaire to determine his reaction to the exam and to 


attempt to determine any other factors which might have 
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influenced his score. The questionaire is presented іп 


Appendix C. 


D. HYPOTHESES 
Based on the premise that recent developments in 
computer-aided testing tools would effectively evaluate 
students, the goal of the experiment was to focus оп 
determining how such tools compare with traditional written 
examinations. More specifically, this experiment was 
designed to evaluate the validity of the Interrogative 
Diagnostic Model. The following hypotheses were tested: 
H1: The scores achieved by a student will be the same 
irrespective of whether the student takes the written 
or computer tests. 


H2: The evaluation quality both of the computer and of 
the written exam will be the same. 


H3: The number of questions presented to the student in 
the computer exam will be the same as the number of 
questions in the written exam. 

H4: The time taken to complete the computer exam will be 


the same as the time taken to complete the written 
exam. 


E. HYPOTHESIS TESTING METHODOLOGY 

In testing the above hypotheses, a factor analysis using 
the student's computer score, section and responses to the 
questionaire was performed to identify dominant factors. A 
regression analysis was done with these factors as 


independent variables against the written exam score as the 
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dependent variable. In testing Hl, the significance level of 
the coefficient of the computer score term was used. In 
addition, the written exam scores were converted to the 
computer scale and a difference in means test between the 
computer and written exam scores was conducted. 

The evaluation quality (H2) for both the written final 
exam and the computer final exam was be measured by 
comparing these scores with the average score of the 
student's two previous written examinations. This was 
considered an appropriate measure because the entire course 
consisted of three examinations with the third and final 
examination being comprehensive having only about 10 percent 
of new material. 

Since the computer examination could theoretically 
present the same number of questions as the written 
examination, it was expected that the number of questions 
presented and time taken for the computer exam would be the 
same as those of the written exam. H3 was tested by counting 
the number of questions in the computer and written 
examinations and the two counts were compared using a 
diffenence in means test. 

In testing H4, the times taken by the student for the 
computer exam and for the written were measured and compared 


using a difference in means test. 
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VI. ANALYSIS OF EXPERIMENT RESULTS 


The computer program automatically saved the exam 
responses, the computed score and the time taken by each 
student. In performing the factor analysis, eight variables 
were included to capture each of the following: 1) the 
computer score (COMPSCOR), 2) the section (SECTION), 
3)inability to explain answers (XPLNANS), 4) lack of essay 
questions  (NOESSYQ), 5) comfort level in using computers 
(COMFLVL), 6) percieved effectiveness (PCVDEFCT), 7) 
perceived pressure (PCVDPRES), 8) general feelings toward 
computer testing (GENLATT). 

The factor extraction technique of principle axis 
factoring with varimax rotation was used. Items which 
failed to demonstrate convergent and discriminate ability by 
loading cleanly ona factor were deleted. The results of 
this analysis are reported in Table A. The items loaded on 
four factors. They were the computer score, the section, 
perceived effectiveness, and general discomfort with the 


concept of computer testing. 
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TABLE 2. VARIMAX ROTATED FACTOR MATRIX 


PEN κ 0 eee eee -  νκ«ν eee eee eee ee eee eee ee Yr 1 T c ee ee ee ee T Y 1 ee р 


Variable/Factor 1 2 3 4 
SECTION -0.32772 0.20560 0.85924 0.09138 
COMPSCOR 0.05631 0.87267 0.21429 -0.12195 
GENLATT 0.88687 0.04518 -0.12319 0.04472 
XPLNANS 0.56898 -0.38634 0.15509 -0.46568 
NOESSYQ 0.65517 -0.07445 0.63430 0.06525 
PCVDPRES 0.79692 0.26031 -0.18654 0.11812 
COMFLVL 0.09884 0.66550 -0.04637 0.35176 
PCVDEFCT 0.10792 0.04534 0.12618 0.91199 


meme шын» фы ЕШР ЧЕК т» ӘР ӘР ӘӘ ӘР ӘР ӘР a ae ee Se Se ee ee ee ee ee ee ee ee eee πα. αν ααν απ μιν eee ee ee απ - - - αν αν αν eee 2 ee eee) απαν ЧОР ЧОР чыт Фын» ЧЫЫР «αν Щз 


Considering only variables with loads above 0.80, it 
can be seen that factor 1 loads on general feelings towards 
computer testing, factor 2 on the computer score, factor 3 
the section and factor 4 on the perceived effectiveness of 
computer-aided exams. 

Using the scores of these four factors as independent 
variables, a multiple regression was run against the the 
written exam scores. The results of this regression are 
shown in Table 3. 


TABLE 3a. REGRESSION OF THE FACTORS AGAINST 
WRITTEN EXAM SCORE 


Фан» бишәр ep ae ee ee eee eee ee ee  - αι. ee ee eee ees eee ee eee eee eee ee S A 


Independent coefficient std. error t-value sig.level 
variable 
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CONSTANT 60.766667 1.359042 44.7129 0.0000 
FACTOR1 0.868266 1.382275 0.6281 0.5356 
FACTOR2 7.751186 1.382275 5.6076 0.0000 
FACTOR3 1.52467 1.382275 1.1030 0.2805 
FACTOR4 -1.42072 1.382275 -1.0275 0.3140 
R-SQ. (ADJ.) = 0.5094 SE=7.443780 

MAE = 5.356040 DurbWat= 1.959 
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TABLE 3b. ANALYSIS OF VARIANCE FOR THE FULL REGRESSION 


Source Sum of Squares DF Mean Square F-Ratio P-value 
Model 890.12 4 472.530 6.52091 .0002 
Error 1385.25 25 55.4099 
Total (Cor.) 3275.37 29 
R-squared = 0.577071 
Stnd. error of est. = 7.44378 
R-squared (Adj. for d.f.) = 0.509403 
Durbin-Watson statistic = 1.95859 
From the table it is evident that factor 2 (computer 


score) is the only significant 


factor. A separate regression 


was therefore run using the computer score as the only 
independent variable. Results of this regression are shown 
in Table 4. 
TABLE 4a. REGRESSION OF COMPUTER SCORE AGAINST 
WRITTEN EXAM SCORE 
Independent coefficient std. error t-value sig.level 
variable 
CONSTANT 32.075889 3.67456 8.7292 0.0000 
COMPSCOR 10.172458 1.251616 8.1275 0.0000 
R-SQ. (ADJ.) = 0.6635 SE = 6.025962 
MAE = 4.970321 DurbWat = 2.076 
TABLE 4b. ANALYSIS OF VARIENCE FOR THE FULL REGRESSION 
Source Sum of Squares DF Mean Square F-Ratio P-value 
Model 2398.63 1 2398.63 66.0556 .0000 
Error 1161.99 32 36.3122 
Total (Corr.) 3560.62 33 
R-squared = 0.673655 R-squared (Adj. for d.f.) = 0.663456 


6.02596 
2.07571 


Stnd. error of est. 
Durbin-Watson statistic 
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The regression was rerun after transforming the written 
and the computer scores to a scale with origin (0,0) so that 
the constant term could theoritically be reduced to Zero. 
For this smallest of the written and computer scores were 
subtracted from the original data. The effect of this 
transformation on the regression are shown in Table 5. 


TABLE 5. REGRESSION OF COMPUTER SCORE AGAINST 
WRITTEN EXAM SCORE AFTER DATA 


TRANSFORMATION 
Independent coefficient std. error t-value sig.level 
variable 
CONSTANT 4.248347 2.498387 1.7004 0.0987 
COMPSCORO 10.172458 1.251616 8.1275 0.0000 
R-SQ. (ADJ.) = 0.6635 SE = 6.025962 
MAE = 4.970321 DurbWat = 2.076 


The computed t-value for the constant term is 1.7004. 
Since this is less than the critical value at 95 percent 
confidence level (1.960), the constant is statistically not 
significant. Thus it was considered appropriate to to 
disregard the constant term and to force the the regression 
line through (0,0). The results of this regression are 


presented in Table 6. 
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TABLE 6a. REGRESSION OF COMPUTER SCORE AGAINST WRITTEN 
EXAM SCORE AFTER DATA TRANSFORMATION AND 
DISREGARDING CONSTANT TERM 
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Independent coefficient std. error t-value sig.level 
COMPSCORO 12.110138 05532355 22.7483 0.0000 
R-SQ. (ADJ.) = 0.9401 SE = 6.196253 

MAE = 5.002530 DurbWat = 1.994 
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Source Sum of Squares DF Mean Square  F-Ratio P-value 
Model 19868 .0 1 19868.0 517.483 .0000 
Error 1266.99 33 38.39 

Total 21135.0 34 


R-sguared = 0.940053 

Stnd. error of est. = 6.19625 
R-squared (Adj. for d.f.) = 0.940053 
Durbin-Watson statistic = 1.99362 
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The the high values of R-squared, the F-ratio and the 
low mean square error tend to confirm that it is sound to 
model the computer score as an important factor influencing 
the written examination score. The difference in means test 
was also performed using the transformed data (Table 7). It 


can be seen that H1 cannot be rejected. 
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TABLE 7. TWO-SAMPLE ANALYSIS RESULTS 


Sample 1 Sample 2 Pooled 
Sample Number of Obs. 34 34 68 
Statistics: Average 2.99149 2.90776 2.94963 
Variance 1.86803 1.7982 1.83311 
Std. Deviation 1.36676 1.34097 1.35393 
Median 2.89474 3:2 3.11316 
Difference between Means = 0.0837214 
Conf. Interval For Diff. in Means: 95 Percent 
(Equal Vars.) Smpll - Smp12 -0.572049 0.739492 66 D.F. 
(Unequal Vars.) Smpll - Smpl2 -0.572054 0.739496 66.0 D.F. 


Ratio of Variances = 1.03884 
Conf. Interval for Ratio of Variances: 0 
Sample 1 v Sample 2 


Percent 


Hypothesis Test for HO: 
Diff = 0 Computed t statistic = 0.254956 
vs Alt: NE Sig. Level = 0.799549 


at Alpha = 0.05 so do not reject HO. 
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The second hypothesis, that the computer exam would 


exhibit the same quality of evaluation as the written exam, 


was tested by comparing both the written exam scores and the 


computer exam scores with the average of the scores of two 
previous exams the students had taken. The rationale for 
this comparison was discussed previously in hypothesis 


testing methodology. The combined results of the two 


previous exams, called STUQUAL, served as the standard in 


determining quality. When the the written and computer 


scores were compared with STUQUAL, the constant term 


generated was not statistically significant and it was 


considered appropriate to disregard the constant and force 


the regression through zero. The results of this regression 
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are shown in Table 8. These results show an R-squared of 
0.9809 for the written final exam and 0.9380 for the 
computer exam. Though there is a difference, it is 
considered slight, and based on these results, H2 cannot be 
rejected. 


TABLE 8a. REGRESSION OF WRITTEN FINAL SCORE 
AGAINST PRIOR AVERAGE SCORE 


--—— qu ap uam uam --ωαω κα κ we ww ww ww ο αν αν ο αν ew ЫН» ew ee eee eee es eee ee ee dump dump quum dump σον 


FINAL 1.426209 0.03463 41.1837 0.0000 
R-SQ. (ADJ.) = 0.9809 SE = 12.437046 
MAE = 10.019345 DurbWat = 1.828 


TABLE 8b. ANALYSIS OF VARIANCE FOR THE FULL REGRESSION 


Model 262353. 1 262353. 1696.10 .0000 
Error 5104.44 33 154.680 
Total 34 267457. 


R-squared = 0.980915 

R-squared (Adj. for d.f£.) = 0.980915 
Stnd. error of est. = 12.437 
Durbin-Watson statistic = 1.82842 
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TABLE 8c. REGRESSION OF COMPUTER SCORE 
AGAINST PRIOR AVERAGE SCORE 


Ind.variable coefficient std. error t-value sig.level 


COMPSCOR 29.259444 1.308931 22.3537 0.0000 
R-SQ. (ADJ.) = 0.9380 SE - 22.407366 
MAE - 17.488222 DurbWat - 1.516 
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TABLE 8d. ANALYSIS OF VARIANCE FOR THE FULL REGRESSION 


ЧЫ шшр ы) ee ο ρω αν αν ο αν ЧЕР «αν ιν ιν ιν ee αν αν αν αν oe «αν «παν απο 


Model 250888. 1 250888. 499.687 -0000 
Error 16569. 33 502.09 
Total 267457. 34 


R-squared = 0.93805 

R-squared (Adj. for d.f.) = 0.93805 
Stnd. error of est. = 22.4074 
Durbin-Watson statistic = 1.51624 
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Hypothesis 3, that number of questions presented to the 
student in the computer exam will be the same as the number 
on the written, was rejected. The written exam consisted of 
75 questions. The number of questions asked by the computer 
ranged from 9 to 60. The average number of questions asked 
was 29. A difference in means test was performed comparing 
the number of questions asked by the computer exam with the 
number of questions in the written exam (Table 9). Results 
of the analysis show that H3 is rejected. 

The fourth hypothesis to be tested was that the time 
taken to complete the computer exam would be the same as the 
time taken to complete the written exam. The times for both 
groups on each exam are shown in Table 10a. Because the 
questions in the computer exam were the same as those on the 
written exam, Section 1 students who took the written exam 
first, had seen all the questions the computer could 
theoretically ask. On the other hand Section 2. students, 


who had taken the computer exam first, had only seen a 
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TABLE 9. DIFFERENCE IN MEANS BETWEEN NUMBER 
OF COMPUTER ASKED QUESTIONS 
AND WRITTEN TEST 


NUMQUEST 
Sample Number of Obs. 34 
Statistics: Average 29.0588 
Variance 181.148 
Std. Deviation 13.4591 
Median 26 
Confidence Interval for Mean: 95 Percent 
Sample 1 24.3616 33.756 33 D.F. 
Confidence Interval for Variance: 0 Percent 
Sample 1 
Hypothesis Test for H0: 
Mean = 75 Computed t statistic = -19.9033 
vs Alt: NE Sig. Level = 0 
at Alpha = 0.05 so reject HO. 


portion of the questions of the written exam. To avoid any 
learning/memory effects that could have occurred between the 
two exams, the written exam time of Section 1 was compared 
with the computer exam time of Section 2. A difference in 
means test was run on these times and the results are shown 
in Table 10b. Based on the results of this test H4 was 
rejected. 

A summary of the results of this experiment are 


presented in Table 11. 


29 


TABLE 10a. AVERAGE EXAM TIMES (minutes) 


Computer Exam Time Written Exam Time 
Section 1 30 112 
Section 2 53 83 
Combined 39 101 


TABLE 10b. DIFFERENCE IN MBANS BETWEEN COMPUTER 
AND WRITTEN EXAM TIMES 


CTIME WTIME Pooled 
Sample Number of Obs. 13 21 34 
Statistics: Average 52.7692 112.095 89.4118 
Variance 236.192 179.89 201.004 
Std. Deviation 15.3685 13.4123 14.1776 
Median 53 120 103.5 
Difference between Means - -59.326 
Conf. Interval For Diff. in Means: 95 Percent 


(Equal Vars.) Smpll - Smpl2 -69.5199 -49.1322 32 D.F. 
(Unequal Vars.) Smpll - Smpl2 -70.0266 -48.6254 22.9 D.F. 


Ratio of Variances = 1.312983 
Conf. Interval for Ratio of Variances: 0 Percent 
Sample 1 v Sample 2 


Hypothesis Test for HO: 


Diff = 0 Computed t statistic = -11.8573 
vs Alt: NE Sig. Level = 3.00426E-13 
at Alpha = 0.05 so reject HO. 
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Hl: 


H2: 


H3: 


H4: 


TABLE 11. SUMMARY OF RESULTS 


The scores achieved by a student will be the same 
irrespective of whether the student takes the written 
or computer tests. 


NOT REJECTED 


The evaluation quality both of the computer and of 
the written exam will be the same. 


NOT REJECTED 

The number of questions presented to the student in 
the computer exam will be the same as the number of 
questions in the written exam. 

REJECTED 

The time taken to complete the computer exam will be 
the same as than the time taken to complete the 


written exam. 


REJECTED 
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VI. N 510 


This thesis presented the Interrogative Diagnostic Model 
and an experiment designed to test its validity. The model 
provides an adaptive mechanism to tailor testing sessions to 
individual students. The results of the experiment indicate 
that the IDM is a reliable model in evaluating student 
performance. The results also show that the IDM effectively 
evaluates students in less time and with less questions than 
a written test. 

The major benefits in using such a testing system of are 
more efficient use of students' and instructors' time and 
less frustration in test taking for both strong and weak 
students. With this type of testing strong students are not 
bored by having to answer questions which are beneath their 
level of learning and hand weak students are not frustrated 
by having to answer questions beyond their capabilities. 

Further Validation testing will be needed before 
widespread use is made of this model. Possible future model 
testing scenarios would be testing the model in a classroom 
situation in which both computer tests and written tests are 
actively used to grade students. The effects that computer 
interface has on student performance is also area of related 


research. 
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One possible area in which examinations based on the IDM 
might prove beneficial is placement testing in industry and 
the military. The IDM might also prove valuable when 
integrated with computer-aided adaptive instruction. 
However used, the model can ultimately prove to be an 
effective substitute for conventional diagnostic methods 


Currently in use. 
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APPENDIX A 
BAYESIAN APPROACH TO THE INTERROGATIVE DIAGNOSTIC MODEL 

In applying Bayesian analysis to the steps of the 
Interrogative Diagnostic Model presented in Section 3, two 
assumptions are necessary [Sivasankaran 1986. The initial 
hypotheses of step 2 are derived by considering these two 
assumptions. The first is that nothing is known about the 
subject and there is a 50-50 chance that the subject is 
average. This possiblity reflects complete uncertianty and 
becomes the uninformed prior probablity py = 0.50. The 
second assumption is that the student has been evaluated, 
either by a human instructor or historical trends, and that 
a better estimate on the subject is available. This better 
estimate is termed pe. The degree of conviction in the 
uninformed versus the estimated assumptions is a:b where a 
and b are weights. Expressed in probability notation this 


becomes 


P(By) a/(atb) 


P(Be) = b/(atb) 


where Buy denotes the strength of belief in the uninformed 
opinion and Be the strength in the belief in the estimated 
opinion. These strengths form the initial hypotheses of 


step 2. 
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Let M be the number of questions asked іп a given topic 
area. The number of questions answered correctly is C and 


each event is denoted E. The mathematical formulation is 


H pu (1 - ρω) 0 

C 

" Ер = 
С 


Bayes theorm P(B/A) = P(B) P(A/B)/(P(B) P) may now Бе 


shown below. 


P(E/By) 


P(E/Be) 


applied as follows: 


P(B,/E) = 


C 


(a/a*b) n ри (1-ру) С + (Ь/а+Ь) И р-ро 


(a/a*b) "| Be epi)! aes 


С С 


апа, 


P(B,/E) = 1 - P(B,/E) 


where P(B,,/E) and P(B,/E) represent the updated hypotheses 
that the student has correctly answered C out of M 
questions. Updating the hypotheses is done by steps 3 and 4 
of the model. Step three asks the student a question and 
step 4 calculates the updated hypotheses based on the 


student's response. 
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Given the updated hypotheises there are two possible 
courses of action. The first is to compare the updated 
value of P(By/E) with upper and lower cutoff criteria. The 
upper cutoff criteria, alpha, represents the minimum 
strength of belief in the estimated opinion that is required 
to conclude that the students knowledge is good. If P(B,/E) 
exceeds alpha, the student is judged knowledgeable enough in 
the topic area being evaluated that no further questions are 
needed. Conversely, the lower cutoff criteria, beta, 
represents the minimum strength of belief in the estimated 
opinion required to discern weak students. If P(B,/E) does 
not exceed beta, the student is judged unknowledgeable in 
the topic area being evaluated and no further questions in 
that topic area are needed. 

The second course of action is necessary when all the 
questions in the critical level have been asked and the 
updated hypothesis P(By,/E) lies between alpha and beta. In 
this case the student is considered to have an average 
knowledge of the subject area and the model then proceeds to 
the lower level of supplementary questions in the same topic 
area or if all the supplemental questions have been asked, 
the model proceeds to a new topic area. 

IDM step 5 compares P(B,/E), the revised estimated 
opinion, with the cutoff values and if neither of the cutoff 
criteria have been been reached the model returns to step 4. 


If one of the criteria has been reached or there are no 
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questions remaining to be asked in the in the cognitive 
level, the model proceeds to step 6. Step 6 accepts or 
rejects the updated hypothesis based on the cutoff criteria. 
If student has exceeded the upper cutoff or has not exceeded 
the lower cutoff, the hypothesis that the student is average 
is rejected. This means that the student has been evaluated 
as either surerior or poor and, in either case, the model 
proceeds to a new topic area by returning to step one. If 
the hypothesis is accepted step 7 will ask further 
supplementary questions to further define the students 
knowledge level or, if the supplenentary questions have been 


asked, will proceed to a new topic area. 
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This 


program 


begins 


names used in the program files. 
IQ1P2.PRG 


implement 


Chapter 


referred 


Figure 3. 


Variable 
aa 


aalfa 


bb 


bbeta 


moreq 


nf 


APPENDIX B 
PROGRAM LISTING 


appendix contains a listing to the dBASE III Plus 
files which implementated the IDM. The appendix 
with a list of the memory variables and datafield 


The programs IQ1P1.PRG and 
are an example of the programming method used to 


the two screen critical questions mentioned in 


IV. The program files using this method are 


to as QUESTION in the program structure chart of 


MEMORY VARIABLES 
Representing 
degree of conviction in uninformed probability 


threshold for beliving subject is above average 


degree of conviction in expert probability 


threshold for beliving subject is below average 
value of combinations returned from factor.prg 


counter in factor.prg 


temporary storage variable for (numasked - right) 


temporary storage variable for k! 


flag indicating if further 
are needed 


supplemental questions 
for numasked! 


temporary storage variable 
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- 


numasked 


numcrit 
numsup 


PBUE 


PBXE 


PEBU 


PEBX 


rf 


right 


SBU 
SBE 


stop 


temp 


wrong 


number of 
topic and 


questions asked thus far of a certian 
type 

number of critical questions of a given type 
number of supplementary questions of a given type 


Bayesian probability of the uninformed opinion 
being correct given the event 


Bayesian probability of 
being correct given the 


the estimated opinion 
event 


probability of the event given the uninformed 
opinion 


probability of the event given the estimated 
Opinion 


temporary storage variable for right! 


number of questions answered from a given type 
and topic area 


strength of belief in uninformed opinion 
strength of belief in estimated opinion 


flag indicating if further critical questions 
are needed 


temporary storage variable used in factor.prg 


number of questions answered incorrectly froma 
Given type and topic area 


uninformed prior probability that a subject 
is not average 


expert's prior probability that a subject is 
above average 
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Field Names 


A 

ALFA 
ANSWER 
AREA 
ASTART 
AFINISH 
B 

BETA 


CUTOFF 


NAME 
LEVEL 
PROGRAM1 
PROGRAM2 


PU 


PX 


QNUM 


QUESTION 


SRESPONSE 


SSN 


SUP 


TYPE 


DATAFIELD NAMES 
Remarks 

degree of conviction in uninformed probability 
threshold for beliving subject is above average 
the answer to a particular question 

the topic area of questions being asked 
the time the subject was presented a question 
the time the subject answered the question 
the degree of conviction in expert probability 
threshold for beliving subject is below average 


minimum percentage of supplemental questions the 
program will present to the subject 


subjects last name 

difficulty level of a particular question 

the first page program of a critical question 
the second page program of a critical question 


uninformed prior probability that a subject 
is not average 


expert's prior probability that a subject is 
above average 


question identifier 


memo entry containing the question and 
multiple choice answers 


the subjects answer to the question 


last four digits of a subject's SSN for 
identification purposes 


name of the DBF file containing the supplemental 
questions 


cognative domain level of question being asked 
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* Program..: MAIN.PRG 

* Author...: PAT ODONNELL 

* Date.....: SPRING 1988 

x Notice...: Copyright (c) 1988, PAT-SIV, A11 Rights 
* Reserved 

* Notes....: THESIS 


SET TALK OFF 

SET BELL OFF 

SET ESCAPE ON 

SET CONFIRM ON 
SET COLOR TO Wt/B 


* This is the main program module for a computer aided 
* testing program. This module calls initilaizes 

* criterion values, and then calls two other modules 
Χ which do the work of the program. 

aa = 0 

bb = 0 

aalfa = 0 

bbeta = 0 

areanum = 1 

X = 0 

U = 0 

ccutoff = 0 


USE criterion 
STORE ALFA TO aalfa 
STORE BETA TO bbeta 
STORE A TO aa 
STORE B TO bb 
STORE PX TO x 
STORE PU TO u 
STORE CUTOFF TO ccutoff 
USE 


SELECT 1 

USE blkboard 
SELECT 2 

USE dbarea 


CLEAR 
TEXT 2 
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WELCOME TO THE 
COMPUTER AIDED TESTING SECTION 


OF THE IS-4183 FINAL EXAMINATION 


ENDTEXT 
@ 22,10 
WAIT 


CLEAR 
DO INFO 


STORE " " TO aarea 

SELECT 2 

GO TOP 

DO WHILE areanum <= 3 
STORE area TO aarea 
DO ASK 
SELECT 2 
areanum = areanum + 1 
SKIP 

ENDDO 


CLEAR 
TEXT 


ENDTEXT 


* EOF MAIN.PRG 
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* Program..: INFO.PRG 

* Author...: PAT ODONNELL 

* Date.....: SPRING 1988 

* Notice...: Copyright (c) 1988, PAT-SIV, All Rights 
x Reserved 

* Notes....: THESIS 


SET TALK OFF 
SET BELL OFF 
SET CONFIRM ON 


x This program module introduces the program to the 

x student,gives instructions on using the program and 

* gets the student's name and last four digits of his 

x SSN for scoring. 

CLEAR 
TEXT 

Good morning and welcome to an experiment in 

computer aided testing. The purpose of this experiment 
is to try and determine whether a computer, using a 
small bank of questions and Bayesian analysis, can 
determine an individual's knowledge level in a given 
subject area as effectively as a traditional written 
exam. The program presented here is not a mere test 
with a fixed number of questions to be answered. This 
program adjusts both the number and type of questions 
to the individual. Should this experiment prove 
successful, testing programs of this type could have 
broad application in educational, industrial and 
military environments. 

ENDTEXT 

@22,10 

WAIT 

CLEAR 
TEXT 
INSTRUCTIONS: You will be shown a series of questions 
and the program will prompt you for the answer. Just 
type in your ansver folloved by a carriage return. Some 
questions are presented in two screens. You may page 


back and forth between screens by following the system 
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prompts. If you have any questions there will Бе 
someone in the lab to answer them. 


IMPORTANT: Be aware that you must answer each question 
in order, in other words you cannot skip a question and 
return to it later. 
Professor Sivasankaran will use the higher score 
between this computer aided test and your written exam, 
so this does "count." 
Good luck! 
ENDTEXT 
@ 22,10 

WAIT 


CLEAR 
SELECT 1 
MNAME - ' : 
MSSN = 0000 
@ 10,10 SAY [Enter your lastname and last four digits 


of your] 

@ 11,10 SAY (social security number in the appropriate 
blocks. ] 

@ 14,10 SAY " LASTNAME: " 

@ 14,33 GET MNAME PICTURE "!!)!!)?)9?!?!9?91!1??!!!1!!1!" 

READ 

@ 16,10 SAY " SSN: " 

@ 16,29 GET MSSN PICTURE "####" 

READ 


APPEND BLANK 
REPLACE NAME WITH MNAME, SSN WITH MSSN 


RETURN 


x EOF INFO.PRG 
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+ + Жж + + + 


Program..: ASK.PRG 
Author...: PAT ODONNELL 


Se : SPRING 1988 


Notice...: Copyright (c) 1988, PAT-SIV, All Rights 


Reserved 


Notes....: THESIS 


SET TALK OFF 
SET BELL OFF 
SET CONFIRM ON 


νεος o» x* 


numsup 


This program asks the student a series of critical 
and complementary questions from the topic database 
flle. The program forwards the results of the 
questioning to calculation modules for bayesian 
analysis. 


= 00 


numcrit = 5 
numasked = 00 


stop 

right 
moreq 
STORE 
STORE 
STORE 
STORE 


"МО! 
= 00 
= ' NO' 
"o" TQ supp 
"C" TO type 

' ! TO start 
' ! TO finish 


SELECT 3 
USE &aarea 


GO TOP 


STORE sup to supp 


DO WHILE .NOT. EOF() .AND. stop - 'NO' 


STORE programl TO pagel 
STORE program2 TO page2 


STORE " " TO response 

goback = "y" 

DO WHILE goback <> "N" 
DO DOIT 

ENDDO 


ϐ 24,05 SAY "Your answer to question " 
Q 24,29 SAY page2 

@ 24,32 SAY " is: fone of a, b, c ...)]" 
@ 24, 59 get response picture '!' 

READ 

ϐ 24,05 SAY " You chose " 
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@ 24,25 SAY response 


Q 24,26 SAY " Press ENTER to confirm " 
@ 24,25 GET response PICTURE '!' 
READ 


STORE TIME() TO finish 


IF response = answer 
right = right * 1 
ENDIF 


SELECT 1 

APPEND BLANK 

REPLACE qnum WITH page2 

REPLACE sresponse WITH response 
REPLACE astart WITH start 
REPLACE afinish WITH finish 
SELECT 3 


numasked = numasked + 1 
DO correct 
SKIP 


ENDDO 


IF moreq = 'YES' 


SELECT 4 
USE &supp 
GO TOP 


DO WHILE .NOT. EOF() 
IF type ='S' 
numsup = numsup + 1l 
ENDIF 
SKIP 
ENDDO 


stop = 'NO' 
numasked = 
right = Q 
STORE ' ' TO suppnum 
GO TOP 


0 


DO WHILE .NOT. EOF() .AND. stop = 'NO' 
IF type = 'S' 


STORE STR(RECNO(), 2, 0) TO suppnum 
CLEAR 
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STORE TIME() TO start 


@ 3,10 
a ", QUESTION 
? 
STORE ' ' TO response 
@ 22,20 SAY "Your answer to question " 
@ 22,44 SAY supp 
@ 22,45 SAY "SQ И 
@ 22,47 SAY suppnum 
Q 22,49 SAY " is " 
@ 22,53 СЕТ response PICTURE "|!" 
READ 
@ 22,10 SAY " You chose " 
@ 22,30 SAY response 
Q 22,31 SAY " Press ENTER to confirm Чч 
a 22,30 GET response PICTURE '!' 
READ 


STORE TIME() TO finish 


IF response = answer 
right = right + 1 
ENDIF 
SELECT 1 


APPEND BLANK 

REPLACE qnum WITH suppnum 
REPLACE sresponse WITH response 
REPLACE astart WITH start 
REPLACE afinish WITH finish 
SELECT 4 
numasked - numasked * 1 
DO correct 

SELECT 4 


ENDIF 
SKIP 


ENDDO 
ENDIF 
RETURN 


* eof ask.prg 
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Program..: IQIP1.PRG 

Author...: PAT ODONNELL 

Date.....: SPRING 1988 

Copyright (c) 1988, PAT-SIV, All Rights 


9X 9 ox »* ot ox 
Z 
О 
et 
P 
Ω 
Ф 


Notes....: THESIS 


Consider a STUDENT file with the following fields used 
in a file processing system: 


Name character 15 
Rank character 6 
Address character 20 
City character 10 
State character 2 
Zip numeric 5 
Salary numeric 5 
Age numeric 2 


One of the records in the STUDENT file contains the 
following values: 


Smith LCDR 202FranklinSt Monterey CA 93940 42000 33 
ENDTEXT 

@ 24,10 

WAIT 

RETURN 


*EOF IQ1P1.PRG 
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* Program..: IQ1P2.PRG 

* Author...: PAT ODONNELL 
* Date.....: SPRING 1988 
* 

* Reserved 

* Notes....: THESIS 

CLEAR 

TEXT 


Notice...: Copyright (с) 1988, PAT-SIV, All Rights 


If the Postal authorities decided to increase the zip 
code length to six digits and the application program was 
so modified, John's salary (based on the existing 


data files) would: 


not change at all 
. would increase by 
would increase by 
would increase by 
would decrease by 
would decrease by 
. None of the above 


Orme an wo wm 


ENDTEXT 
RETURN 


*EOF IQ2P2.PRG 
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$3 
$100,003 
$30,000 
$21997 
$40000 
is true 


Program..: DOIT.PRG 


x 

* Author...: PAT ODONNELL 

* Date.....: SPRING 1988 

* Notice Copyright (с) 1988, PAT-SIV, 
* Reserved 

* Notes....: THESIS 


SET TALK OFF 
SET BELL OFF 
SET CONFIRM ON 


* This is the program module which 

* critical questions to the student. 
STORE " " TO goback 

STORE TIME() TO start 

DO &pagel 

DO &page2 

@ 24,5 SAY "Do you want to go back to the previous 

(Y or N)" 

@ 24,59 СЕТ goback picture "!" 

READ 
RETURN 


*eof DOIT.PRG 
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A11 Rights 


presents the 


page? 


Program..: CORRECT.PRG 

Author...: PAT ODONNELL 

Date.....: SPRING 1988 

Notice...: Copyright (c) 1988, PAT-SIV, All Rights 
Reserved 

Notes....: THESIS 


X* X OX o x 


SET TALK OFF 
SET BELL OFF 
SET CONFIRM ON 


* This program module calculates the Bayesian 
х probabilities of the student's correct answers. 
CLEAR 

£ = 0 

eval = 0 


do FACTOR with numasked, right, f 


SBU = aa/(aa + Db) 

SBX = bb/(aa + bb) 

РЕВО = £ * (u ** right) * ((1 - п) ** (numasked - right)) 
PEBX = £ * (x κκ right) * ((1 - x) κκ (numasked - right) ) 
PBUE = (SBU * PEBU) / ((SBU * PEBU) + (SBX * PEBX)) 

PBXE = 1 - PBUE 


*814,10 say "PBXE =" 
*@14,18 say PBXE 


*a20,10 
*WAIT 
CLEAR 
DO CASE 
CASE type - 'C' .AND. PBXE >= aalfa 
stop - 'YES' 
eval = 1 
DO tally 
CASE type = 'C' .AND. PBXE <= bbeta 
stop = ‘YES! 
eval = 5 
DO tally 
CASE type = 'C' .AND. numasked = numcrit 
moreq - 'YES' 
CASE type - 'S' 


IF (numasked/numsup) >= ccutoff .AND. PBXE >= aalfa 
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stop = 'YES' 
eval = 2 
DO tally 
ENDIF 
IF (numasked/numsup) >= ccutoff .AND. PBXE <= bbeta 
stop = 'YES' 
eval = 3 
DO tally 
ENDIF 
CASE type = 'S' .AND. numasked = numsup 
eval = 4 
DO tally 
ENDCASE 
RETURN 


eof correct.prg 
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* Program..: TALLY.PRG 

* Author...: PAT ODONNELL 

* Date.....: SPRING 1988 

* Notice...: Copyright (c) 1988, PAT-SIV, А11 Rights 
* Reserved 

* Notes....: THESIS 


SET TALK OFF 
SET BELL OFF 
SET CONFIRM ON 


* This is the program module vhich appends the area 
* evaluations to the blkboard file. 
SELECT 1 


APPEND BLANK 

REPLACE name WITH aarea 
REPLACE ssn WITH eval 
SELECT 3 

RETURN 


* EOF TALLY.PRG 
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Program..: FACTOR.PRG 

Author...: PAT O'DONNELL 

See σόν το : SPRING 1988 

Notice...: Copyright (c) 1988, PAT-SIV, All 
Reserved 

Notes....: THESIS 


жх жх ж ж 
J 
fy 
et 
(v 


PARAMETERS numasked, right, f 


SET TALK OFF 
SET BELL OFF 


temp=1 
i = 1 


do while i <= numasked 
temp = temp * i 


i=i+i1 
enddo 
nf = temp 
temp = 1 
i = 1 


do while i <= right 
temp = temp * 1 


і = і +1 
enddo 
rf = temp 
temp = 1 
i 1 


k numasked - right 


do while i <= k 
temp = temp * i 
Weri үс 

enddo 

КЁ = temp 


f = nf/(kf * τε) 


* EOF FACTOR.PRG 
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APPENDIX C 
QUESTION LISTING 

This appendix lists the 75 questions contained in the 
written and computer final exams. The questions are divided 
into the three topic areas: introduction, normalization and 
management. Each topic area begins with five critical 
questions followed by twenty supplementary questions. The 
difficulty level of the question is indicated by a number 
from one to three. This difficulty level was determined by 
the expert opinion of the instructor based on his experience 
of the questions in previous quarters. The meaning of those 


numbers is as follows: 


1 less difficult 
2 average diffculty 
3 very difficult 


The answer to each question is indicated by > or ^. 


55 


INTRODUCTION QUESTIONS 


Introduction Critical Question 1 


Level 3 


Consider a STUDENT file with the following fields used 


ina file processing systen: 


Name character 
Rank character 
Address character 
City character 
State character 
Zip numeric 
Salary numeric 
Age numeric 


15 
6 
20 


р 
NUN © 


One of the records in the STUDENT file contains the 


following values: 


Smith LCDR 202FranklinSt Monterey CA 93940 42000 33 


If the Postal authorities decided to increase the zip 
code length to six digits and the application program was 
so modified, John's salary (based on the existing 


data files) would: 


a. not change at all 
b. would increase by 
c. would increase by 
d. would increase by 
^ e. would decrease by 
f. would decrease by 
g. None of the above 
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$3 
$100,003 
$30,000 
$21997 
$40000 
is true 


Introduction Critical Question 2 Level 2 


Which one of the following portrays accurately a subset 
of the functional components of a DBMS? 


К, 
| | Logical 
| Definition | I/O request 
l | | Tools sz ----------- > | | 
i User | -->-- | Subsystem | | | 
| or | | | | DBMS | 
|Application| | engine | 
| Program | | | 
| --»-- | | Logical | | 
| Processing | I/O request | | 
| Interface |<-------------- » | | 
| Subsystem | 
| | 
II. 
| раа | 
| Design | <---------- | 
| Subsystem | | | | 
| | Logical | | DBMS | Physical 
I/O |------ >| {<---> 1/0 
requests | | engine | request 
| Data | | | | 
| Administration | | | | 
| Subsystem | <---------- | 
| | 
III. 
| | Logical | | Physical 
| Data | I/O Request | DBMS | I/O request 
| Dictionary |4-------------- >| engine | 
| | | | <-------- > | | 
| | | | | Op | 
ISystem| 
a. I and II only | --------- >| | 


| | 
b. II and III only | DB | 1/0 
> c. I and III only | | Commands 
а. III only 
e. all the above 
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Introduction Critical Question 3 


The Prime Area of a storage that uses index 


Level 3 


sequential 


method contains the following records (in unblocked format): 


Track 1 
Track 2 
Track 3 


index would look like 


If REC 


REC 5 REC 10 
REC 25 REC 30 
REC 45 REC 50 


REC 
REC 
REC 


22 and REC 23 were to be added, 


a. 20 20 null b. 1 20 22 
2 40 40 null 2 40 40 
3 60 60 null 3 60 60 
νο N 20 20 null a. "I 20 22 
2 30 40 address of 35 2 40 35 
3 60 60 null 3 60 60 
e. 1 20 20 address of 22 
2 40 40 address of 33 
3 60 60 null 
Introduction Critical Question 4 
Consider the following object diagram: 
COMPUTER STUDENT 
| Computer# | | Studenti | | 
| Osver | | Sphone | | 
| Acqdate | | CLASS (MV) | | 
| Status | | APPOINTMENT(MV) | | 
| Station# | | | 
| SOFTWARE (MV) | 
| | SOFTWARE 
| 
| Sname | | 
| Copy | | 
| APPOINTMENT (MV) | | 
| COMPUTER (MV) | | 
| | | 
| 
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15 КЕС 
35 КЕС 
55 REC 


20 
40 
60 


the new track 


address of 
null 
null 


address of 
address of 
null 


Level 1 


CLASS 


Cname 
Tothrs 
STUDENT ( MV) 


APPOINTMENT 


Apptdate 
Apptstrt 
CLASS 
STUDENT 
COMPUTER 
SOFTWARE ( MV) 


22 


22 
35 


The Bachman diagram for the above objects is given by: 


I. E os > COMPUTER <<---------- >> SOFTWARE <<---| 


|-------------- >> APPOINT <<------------------ | 
| ----------- >> MENT <---------- | 


nma |------ » COMPUTER ««---------- >> SOFTWARE <<---| 


|---------------- > APPOINT <<------------------ | 
|------------ >> MENT  ««---------- | 


τν STUDENT < <-====- >> CLASS <---| 


III. lass ZEGUMPULEN <<-----=---- >> SOFTWARE <<---| 


|---> STUDENT <<------- >> CLASS <--- | 
a. only I is true d. only II and III are true 
b. only II is true e. all are true 
> c. only III is true f. none are true 
Introduction Critical Question 5 Level 1 


Consider the following table: 


Scheduling Inventory Usage Analysis 
Students X X 
Class Rosters X X 
Computers X 
Faculty X 


Which of the folloving is true? 
r. Column I indicates the different applications 
in a database system and the remaining columns 
indicate files used by a DBMS. 


DI. "X" indicates vhich file is used by which DB 
application. 
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Each column in the table is called a "schema". 


Column 1 indicates the different files and the 
remaining columns indicte applications. 


The table is called a "schema". 


I, II, IV and V only 
II, IV and V only 

II and IV only 

II, III and IV only 
All the above are true 


Φ ΩΩ ζῶ 
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Introduction Supplementary 


Question 1 


Level 2 


The disadvantages of file processing systems are: 


T data is separated and isolated 
II. data is often duplicated 
III. application programs are dependent 


formats 


on 


IV. processing is slower than manual systems 


a. all the above 


> b. I, II and III only 
c. II, III and IV only 


d. I and III only 


Introduction Supplementary Question 2 


The functions of a DBMS engine include: 


I. build conceptual schema structure 


II. manage disc and 


memory space 


III. provide object and domain definitions 
IV. map logical I/O requests into physical activity 


a. I and III 
> b. II and IV 
c. I, II and IV 
d. all of the above 


e. none of 


Introduction Supplementary 


Whether a property is 
nothing to do with whether 
nonobject property. 


Introduction Supplementary 


Database data can be 
user: 


TI entering data 
II. entering data 
III. entering data 
IV. entering data 


the above 


Question 3 


file 


Level 3 


Level 3 


Single- or multiple-valued has 


it is an object or 


T or F 


Question 4 


updated in different 


via a load utility 
with a form 

in a tabular format 
through database views 
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Level 2 


ways 


by 


a 


a. all of the above 
b. II, III and IV 

> C. I, II and III 
d. I, II and IV 


Introduction Supplementary Question 5 Level 2 
Some of the functions of a DBMS are: 


I^ store, retrieve and update data 

II. provide object and Bachman diagrams 

III. provide coordination and control facilities for 
concurrent processing 

IV. provide facilities for backup and recovery 


a. all the above 
b. II and IV 
c. I, II and IV 
> d. I, III and IV 
Introduction Supplementary Question 6 Level 3 


Integrity rules are very easy to enforce using a DBMS 


T or F 


Introduction Supplementary Question 7 Level 3 


The domain of an object property is a set of object 
instances. 


T or F 


Introduction Supplementary Question 8 Level 3 


The fact that a property has multiple values does not 
imply that it must be an object property. 


T -or F 


Introduction Supplementary Question 9 Level 2 


An OBJECT is a named collection of determinants that 
sufficiently describes an entity in the users work 
environment. 

T or F < 
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Introduction Supplementary Question 10 Level 2 

A set of attributes with the property that no two tuples 
in the relation can ever have the same values of these 
attributes is called the relation key. 


T or F 


Introduction Supplementary Question 11 Level 2 


The Portion of an object that is visible to a particular 
application is called a VIEW. 


T or F 


^ 


Introduction Supplementary Question 12 Level 2 

A complex relationship between two relations can be 
represented by including the key of one relation as an 
attribute in the other. 


T or F 


Introđuction Supplementary Question 13 Level 1 


The components of a database application are: 


I. hardware 
II. programs 
III. data 


IV. procedures 
V. personnell 


> a. all the above 
b. I, II, III and V 
C. I, II and III 
d. III, IV and V 


Introduction Supplementary Question 14 Level 1 
Some types of database data are: 
F: application data 
II. database metadata 


III. overhead data 
IV. transaction log data 
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a. all of the above 

> bD. I; II and її 
C. I and III only 
d. II and IV only 

Introduction Supplementary Question 15 Level 1 
The different phases of database system development are: 

D definition 
II. requirement 
III. evaluation 


IV. design 
V. implement 


a. I, II, III and IV 
b. I, III, IV and V 
C. IV and V 
> d. all of the above 
Introduction Supplementary Question 16 Level 1 


Different attributes may take values from the same 
domain. 


T or F 


Introduction Supplementary Question 17 Level 1 
Each record ina relation is unique. 


T or F 


Introduction Supplementary Question 18 Level 1 
A relation ís a flat file. 


TOITE 


Introduction Supplementary Question 19 Level 1 


To be able to join two relations there must always be 
one common attribute. 


T or F 
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Introduction Supplementary Question 20 Level 1 

The SELECT operation defines a subset of tuples of a 
given relation (i.e. horizontal slice) based on a Boolean 
condition. 


T or F 
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NORMALIZATION QUESTIONS 


Normalization Critical Question 1 Level 3 


Consider the relation with the following structure: 


STUDENT_USAGE (SID, Computer_Number, Class, Sname, Total_Hours) 


You are also given: SID ----- >> Computer Number 
эо === 55 >> Class 
SID- === > Sname 
ρω -᾽ > Total_Hours 
Sname ---> SID 


Which of the following is false? 
г. SID and Sname are determinants but SID alone is the key 


II. There is at least one transitive dependency leading to 
anomolies 


III. There is exactly one MVD 


IV. The relation can be upgraded to the next higher normal 
form by decomposing it into smaller relations 


I, II and III 

II, III and IV 

I, II and IV 

all the above 
none of the above 


ῷ ΩΩ σΩ 
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Normalization Critical Question 2 Level 3 

Consider the following three relations with the given 
functional dependencies. Identify the normal form in which 
each relation is in. 


R1 (A, C, B, D) A ---> B 
R2 (A, B, E) AC ---> D 
R3 (A, C, D, F) А ---> Е 
B ---> CD 
D ---> F 
a. Ri - 2ndNF R1 - l1stNF 
R2 - 3rdNF R2 BCNF 
R3 - 2ndNF R3 2ndNF 
C. Ri - 3rdNF R1 istNF 
R2 - BCNF R2 J3rdNF 
R3 - 2ndNF R3 2ndNF 


e. none of the above 


Normalization Critical Question 3 Level 3 


Consider the relation S J T with three objects respectively; 
Student, Subject and Teacher. The following constraints apply: 


-- for each subject, each student of that subject is 
taught by only one teacher 


-- each teacher teaches only one subject 
-- each subject is taught by several teachers 


Which of the following is true? 


τ; S J T is in BCNF 

II. S_J_T has MVDs 

III. S_J_T has one possible composite key 
a. I only d. all of the above 
b. II only ^ e. none of the above 


C. III only 
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Normalization Critical Question 4 Level 1 
When these two relations are joined, 
PART SUPPLIER 
P# COLOR CITY S# P# 
Pl red LA Sl Pl 
P2 blue NY 51 Р2 
P3 green SF S2 Pl 
P4 blue NY 
Which one of the following is true? 
a. there will be exactly 12 tuples in the resultant join 
> b. there will be exactly 3 tuples in the resultant join 
c. there vill be exactly 4 tuples in the resultant join 
d. there will be exactly 7 tuples in the resultant join 
e. none of the above 
Normalization Critical Question 5 Level 1 
Given the relations below: 
CABLE CUSTOMER CONVERTOR 
| | | | | | 
Figure 1 | Account& |--0O----- O--| serial# |...| Accountiàé | 
| | | |. - | | 
SHIPMENT PARCEL 
| | | | | | 
Figure 2 | Container& |--|--O--| Parcel& |...| Container | 
| | \ | ΓΙ | 
PART BIN LOCATION 
| | | | | | | 
Figure 3 | Part# 1...] Bing |--0------ |---| Bin#® 1...1 
| |---- | | |] 
PLAYER POSITION 
| | | | | | 
Figure 4 | Name |... .)-=|=>=325 eee |--| Position 1... | 
| | | / \ | | | 


Which of 


ү. 
II. 


Fig 1 
Fig 2 


III. Fig 3 


IV. 


Fig 4 


the following is true? 


represents an optional-optional relationship 
represents a mandatory-optional relationship 
represents an optional-mandatory relationship 
represents a mandatory-mandatory relationship 


a. I and IV only 

b. II and III only 

c. I, II and III only 

d. I, II and IV only 

e. all the above are true 
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Normalization Supplemental Question 1 Level 3 


A loss projection occurs when 


> a. false data is generated by a projection 
b. a projection loses attribute values 
c. a projection loses tuples 
d. a projection cannot be made on two specific 
relations 
Normalization Supplemental Question 2 Level 3 


Which one of the following is true in one to one 
relationships? 


I. Attributes that have a one to one relationship must 
occur together in at least one relation. (call the 
relation R and the attributes A and B) 

II. Both A and B must be part of the key of R. 


III. An attribute can be added to R if it is functionally 
determined by A or B. 


IV. A and B must occur together in R, but should not occur 
together in other relations. 


a. I, II and III 
> b. I, III and IV 
C. II and III 
d. all of the above 
Normalization Supplemental Question 3 Level 3 


When evaluating database design 


a. machine efficiency has highest priority 
b. elimination of modification anomalies has the 
highest priority 
с. relation independence has the highest priority 
> d. priorities will be different depending upon 
requirements 


Normalization Supplemental Question 4 Level 2 


Primary keys must be unique within a relation but 
candidate keys do not have to be unique within a relation. 


T or F < 
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Normalization Supplemental Question 5 Level 3 
In the following functional dependancy A -> B -> C, 


a. the dependancy is a multivalued dependancy 
> b. C is a transitive dependancy with A as its 
determinant 
C. Cis the determinant 
d. A and C are not dependent 


Normalization Supplemental Question 6 Level 2 
If a relation is in domain key normal form, it 
> a. could have no modification anomalies 
b. may still have transitive dependencies 
c. could still have join dependencies 
d. could still have multivalued dependencies 


Normalization Supplemental Question 7 Level 3 


A relation in first normal form can generally be placed 
in second normal form by 


> a. projections 
b. joins 
c. selections 
d. products 
Normalization Supplemental Question 8 Level 3 


An interrelation constraint exists when 


> a. two or more relations share the same attribute 
b. two attributes in the relation have the same 
determinant 
C. two determinants ina relation have the same 
subject attribute 
d. two or more attributes of a candidate key have 
different determinants 


Normalization Supplementary Question 9 Level 3 


À modification anomaly occurs when change to one entity 
causes a change to a second entity. 


T or F < 
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Normalization Supplemental Question 10 Level 3 


Multivalued dependencies occur when 


a. several attributes are dependent upon the 
same determinant 
b. a single attribute has several determinants 
C. there are several transitive dependencies 
> d. several independent attributes are dependent 
on one other attribute 
Normalization Supplemental Question 11 Level 3 


Which of the following is not a relational design 
evaluation criterion: 


a. elimination of modification anomolies 
b. relation independence 
>» C. ease of use 
d. reduction of key attributes 
Normalization Supplemental Question 12 Level 3 


Attributes that have a many to many relationship can 
exist together in a single relation (although this would 
create redundancies). Assume two such attributes E and F, 
residing together ina relation T. 

Which of the following is false? 

I The key of T must be E 

II. The key of T must be F 

III. Both E and F can be candidate keys 


IV. An attribute can be added to T if it is determined 
by E or F 


all the above 


a 
b 

c. III and IV only 
d. 

e. none of the above 
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Normalization Supplemental Question 13 Level 1 


A relation is in domain/key normal form when 


a. the domain of the key is the function of the key 
b. the key is the function of the domain 
c. it has no multivalued dependencies 
› а. every constraint is a logical consequence of 
the definition of keys and domains 
Normalization Supplemental Question 14 Level 1 


If a relation is in third normal form, it must be in 
second normal form. 


T or F 


Normalization Supplemental Question 15 Level 1 


A relation is in Boyce-Codd normal form if 


a. it has only one key 
b. it has no transitive dependencies 
> с. every determinant is a candidate key 
d. there are no deletion anomalies 
Normalization Supplemental Question 16 Level 1 


If a tuple cannot be inserted into a relation until an 
additional fact about another entity is known, it is 
referred to as: 


a. a deletion anomaly 
> b. an insertion anomoly 
c. a functional dependency 
d. an insertion redundancy 


Normalization Supplemental Question 17 Level 1 


If a relation meets the criterion that all попкеу 
attributes are dependent upon all of the key, it must be in: 


second normal form 
fourth normal form 
domain/key nornal form 
Boyce-Codd normal form 


> 


опсо 


TS 


Normalization Supplemental Question 18 Level 1 


In the following functional dependency A -> B -> C, the 
determinant(s): 


a. 1з А only 
b. is C only 
> c. are A and B 
d. are B and C 
Normalization Supplemental Question 19 Level 1 


Multivalued dependencies always exist in pairs. 


T or F 


Normalization Supplemental Question 20 Level 1 
If an attribute is a key it must be a determinant. 


T or F 
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MANAGEMENT QUESTIONS 


Management Critical Question 1 


You are given this transaction log at the 


database crashed. 


Relative 

Record 

Number 

1 οτι 0 2 11:42 START 

2 οτι 1 4 11:43 MODIFY  CUST100 
3 OT2 0 8 11:46 START 

4 OT1 2 5 11:47 MODIFY SP AA 

5 OT1 4 7 11:47 INSERT ORDER 11 
6 CT1 0 9 11:48 START 

7 OT1 5 0 11:49 COMMIT 

8 OT2 2 0 11:50 COMMIT 

9 στι 6 10 11:51 MODIFY SP AA 
10 CT1 9 0 11:51 COMMIT 


(old value) (new value) 


(old value) (new value) 


(old value) (new value) 


Which of the transactions will be reprocessed after the 


last save? 


I. OT1 
LI. OT2 
ΤΙ CTI 

I only 

II only 

III only 


all of the above 
none of the above 


ооп осо 
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Management Critical Question 2 


There are three users ina multiuser 


Level 2 


DB. Each wants to 


perform the following sequence of events. 


USER A USER B 
Read rec 100 Read rec 200 
Change rec 100 Change rec 200 
Write rec 100 Write rec 200 


Which of the following sequences violate 


Te II. 

Read 100 for A Read 200 for B 
Read 200 for B Read 100 for A 
Change 100 for A Change 100 for A 
Write 100 for A Write 100 for A 
Read 200 for C Change 200 for B 
Change 200 for B Read 200 for C 
Write 200 for B Change 200 for C 
Change 200 for C Write 200 for C 
Write 200 for C Write 200 for B 


Select your answer: 


> a. I and II only 
b. II and III only 
c. I and III only 
d. III only 
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Read rec 200 
Change rec 200 
Write rec 200 


s serializability? 


III. 
Read 200 for C 
Change 200 for C 
Read 100 for A 
Write 200 for C 
Change 100 for A 
Read 200 for B 
Change 200 for B 
Write 100 for A 
Write 200 for B 


Management Critical Question 3 Level 3 


Five records are stored physically on a disc track as 
shown below: 


RRN 4 
| | 
/ \ 

RRN 5 / \ RRN 3 
\ 
uc NM 
| (8 νΕΥΥ i 
| crude disc track) | RRN 2 

| | 
| pe 
N / 
N / 
N / RRN 1 
VE 


The actual contents of each record is given in the folloving 
table: 


RPN SUPNAME PARTID PRICE 
l Topnotch P003 9.95 
2 Ugo P014 0.99 
3 Bestparts P020 29: 35 
4 Acme P008 99.99 
5 Gotcha P002 11.00 


Assume you access the data in the following ways: 


Т; Access the entire file in the order specified by a 
linked list created according to ascending order 
(alphabetic) by Supname. 


II. Access the entire file in the order specified by a 
linked list created according to ascending order 
(numeric) by PartID. 


Assuming the disc rotates in a clockwise direction, 
which of the following is true? 


> a. There will be no difference in the number of disc 
rotations if the entire file were to be accessed by 
either I or II. 

Method I will be faster by one rotation than method II. 
. Method II will be faster by one rotation than method I. 
Method I will be faster by two rotations than method II. 
Method II will be faster by two rotations than method I. 


ΦΩΩΣσ 


17 


Management Critical Question 4 Level 2 
The following table is an example of 
PATIENT Record 


Subjects Who Know Subjects Who Know 


Password SICK Password PAIN 
Read Y Y 
Insert Y N 
Actions: Modify Y N 
Delete Y N 
Grant N N 


Fernandez, Summers, Wood model of database security 
. subject-orientated security 

entity-orientated security 

. all of the above 


ν 
ano ow 
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Management Critical Question 5 


You are given the following table: 


Relative 
Record 


Number 


1 


un & W AN 


Student- 
number 


200 
100 
300 
200 
300 
100 


Class- 
number 


Le 


Semester 
88S 
89F 
89F 
88S 
88S 
88S 


vel 1 


Determine which one is the correct inverted list (or index) 


Student- 
number 
100 
100 
200 
200 
300 
300 


I. 


Relative 
Record 
Number 


2 


NW he DH 


υπ. 
Relative 
Class- Record 
number Number 
20 3 
20 6 
30 2 
30 4 
70 1 
70 5 
a. I only 
b. II only 
C. III only 
d. I and II only 
» e. all the above 
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I 


Semester 
88S 
88S 
88S 
88S 
89F 
89F 


II. 


Relative 
Record 
Number 


Wh OY Ut | е 


- 


Management Supplemental Question 1 Level 1 


A binary relationship involves at least two record 
types. 


T or F 


Management Supplemental Question 2 Level 3 


When the relationship is a 1:n and is mandatory in both 
directions, there is a strong likelyhood that the records 
are describing different aspects of the same object. 


T or F 


Management Supplemental Question 3 Level 3 
An object relationship is one-to-one if: 


I. object A contains object B as a single-valued 
object property 

II. object B contains Object A as a single- 
valued object property 

III. object B does not contain object A 


only condition I is necessary 

conditions I and II alone are sufficient 
conditions II or III alone are sufficient 
conditions I, II and III are all necessary 


м 
опоо 


Management Supplemental Question 4 Level 3 


If A is related to B and B is not related to A, it is 
referred to as 0:1 relationship. 


T or F 
Management Supplemental Question 5 Level 3 
The multi-valued attributes in a composite object never 
represent another object. 
T or F 


Management Supplemental Question 6 Level 3 
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The keys of two relations having a compound-object 
relationship between them do not have a common attribute. 


T or F 


Management Supplemental Question 7 Level 1 
An atomic transaction is 


a. any set of activities that operate on the same 

data-items 
> b. a set of activities that form one logical unit of 

work 

c. any set of activities that can be performed with 
one instruction to the DBMS 

d. any set of activities that can be performed 
concurrently 


Management Supplemental Question 8 Level 2 
To roll forward, 


> a. restore the database from a save and apply after 
images 
b. use the current copy of the database and apply 
the after images 
restore the database and apply before images 
use the current copy of the database and apply 
before images 


ΩΩ 


Management Supplemental Question 9 Level 3 
A check point record is written 


a. after all updates have been made to the database 
b. after all requests have been written to the 
transation log 
» c. after all requests have been written to the 
database and the transaction log 
d. before updates have been written to the database 
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Management Supplemental Question 10 Level 1 


In the Fernandez, Summers, and Wood Model of database 
security, a subject can be 


> a. the program that will be processing the database 
b. the element of the database that will be protected 
c. the processing permissions given to a user 
d. the user's password 
Management Supplemental Question 11 Level 2 


Constraints on user's activity can be enforced by 


a. subject-oriented security 
b. object oriented security 
C. password matrices 
› d. user exits 
Management Supplemental Question 12 Level 1 


Functions of a DBA include: 


I; Management of data activity 

II. Management of database structure 
III. Database Personnel and Placement 
IV. Management of the DBMS 


a. I, II, IV only 
b. II and IV only 
> С. All of the above 
d. IV only 
Management Supplemental Question 13 Level 1 


Concurrent processing implies that the DBMS processes 
the queries of various users simultaneously. 


T or F 


^ 


Management Supplemental Question 14 Level 2 

Serializability ensures that results of two concurrent 
transactions are the same as they would have been had they 
been processed one at a time. 


›У Тот Е 
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Management Supplemental Question 15 Level 1 


Database changes that have been made by a transaction in 
process are said to be uncommitted. 


T or F 


Management Supplemental Question 16 Level 1 


Concurrent processing does not lead to lost updates if 
transactions are only reading data. 


T or F 


Management Supplemental Question 17 Level 1 


Deadlock occurs when each of two transactions is waiting 
for a resource the other has locked. 


T or F 


Management Supplemental Question 18 Level 1 


Checkpoints are snapshots of updated parts of а 
database. 


T or F 


Management Supplemental Question 19 Level 1 


The Performance Monitor maintains the configuration 
control of the DBMS. 


T or F 


Management Supplemental Question 20 Level 1 

Interleaving means that vhile two or more transactions 
are processing concurrently the computer system  executes 
some instructions from one, then executes some from the 
other, switching back and forth between then. 


T or F 
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APPENDIX D 
COMPUTER-AIDED TEST QUESTIONNAIRE 


LAST NAME LAST FOUR DIGITS OF SSN 


1. What are your general feelings toward this computer 
aided testing procedure? 


very somewhat somewhat very 
negative negative indifferent positive positive 
Z5 How do you feel that not being able to explain your 


answers affects the accuracy of the test? 


very somewhat somewhat very 
negatively negatively indifferently positively positively 


d How do you feel about not having to answer essay 
questions? 
very somewhat somewhat very 


negatively negatively indifferently positively positively 


4. Did you feel more pressure with the computer test than 
you have usually felt during other vritten exams? 


Yes or No 


D Rate your comfort level in using computers. 
very indifferent very 
uncomfortable comfortable 
somewhat somewhat 
uncomfortable comfortable 
6. Do you think a computer aided exam tests the knowledge 


of the student more effectively than a written exam 


Yes or No 


8 4 


Do you think that making such programs available for 
practice would help students learn subject matter 
better? 


Yes or No 


Please note any comments, likes/dislikes ΟΥ 
recommendations you have about this test or the 
concept of computer aided testing in general? 
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APPENDIX E 
EXPERIMENT DATA 


STUDENT SECTION ЕХАМ1 ЕХАМ2 FINAL COMPSCOR 
1 1 40 43 56 2.34 
2 1 49 47 76 4 
3 1 48 43 59 3.34 
å 1 46 46 46 2 
5 l 50 42 51 2 
6 1 46 41 59 3 
7 1 49 45 64 3.34 
8 1 39 41 64 2.67 
9 1 42 48 74 Е 

10 1 47 47 60 2.34 
11 1 45 45 74 3 
12 1 44 37 45 2 
13 1 48 44 70 3 
14 1 48 47 67 3.34 
15 1 45 46 58 2 
16 1 47 45 74 3.67 
17 1 49 43 68 4 
18 1 46 40 52 2 
19 1 43 43 38 1 
20 1 33 45 ӘӘ 1.67 
21 1 34 38 45 1.67 
22 2 47 48 12 3.34 
23 2 43 42 57 2.34 
24 2 48 49 65 4.34 
25 2 44 46 75 3.34 
26 2 46 43 60 3 
27 2 39 45 55 3.34 
28 2 44 44 61 3.67 
29 2 43 46 51 2.34 
30 2 48 48 72 3.34 
οι 2 49 46 75 3.34 
32 ? 43 41 68 3.34 
33 2 38 43 51 2.34 
34 2 39 43 48 1.34 
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STUDENT  QUEST7 CTIME WTIME NUMQUEST 


(MIN) (MIN) 

T 1 46 120 29 

2 1 28 120 45 

3 1 12 120 17 

4 1 27 120 18 

5 0 18 75 19 

6 1 32 107 25 

7 1 35 119 45 

8 0 19 103 21 
9 1 15 120 33 
10 1 40 120 26 
11 1 55 120 60 
12 l 25 114 41 
13 1 26 т] 35 
14 1 29 104 23 
15 1 22 120 40 
16 0 41 120 46 
17 T 25 90 38 
18 l 19 85 20 
19 0 17 120 9 
20 1 45 120 27 
21 1 56 120 26 
22 1 63 100 17 
23 26 80 9 
24 1 53 70 25 
25 0 48 85 52 
26 0 70 LIS 60 
27 1 44 95 20 
28 1 61 94 29 
29 1 my 53 27 
30 1 56 78 24 
31 0 67 112 36 
32 1 50 70 17 
33 44 60 9 
34 l 27 70 20 
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